{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T07:33:12Z","timestamp":1783668792766,"version":"3.55.0"},"reference-count":222,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,11,11]],"date-time":"2024-11-11T00:00:00Z","timestamp":1731283200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"STI 2030, \u201cBrain Science and Brain-like Research\u201d Major Project","award":["2021ZD0201405"],"award-info":[{"award-number":["2021ZD0201405"]}]},{"name":"NTU RSR and Start Up Grants"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>The field of deep learning has witnessed significant progress in recent times, particularly in areas such as computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models trained on vast amounts of data holds immense promise for practical applications, enhancing industrial productivity and facilitating social development. However, it suffers extremely from the unstable training process and stringent requirements of computational resources. With the increasing demands on the adaption of computational capacity, though numerous studies have explored the efficient training field to a certain extent, a comprehensive summarization\/guideline on those general acceleration techniques of training large-scale deep learning models is still much anticipated. In this survey, we present a detailed review of the general techniques for training acceleration. We consider the fundamental update formulation and split its basic components into five main perspectives: (1) \u201cdata-centric,\u201d including dataset regularization, data sampling, and data-centric curriculum learning techniques, which can significantly reduce the computational complexity of the data samples; (2) \u201cmodel-centric,\u201d including acceleration of basic modules, compression training, model initialization, and model-centric curriculum learning techniques, which focus on accelerating the training via reducing the calculations on parameters and providing better initialization; (3) \u201coptimization-centric,\u201d including the selection of learning rate, the employment of large batch size, the designs of efficient objectives, and model average techniques, which pay attention to the training policy and improving the generality for the large-scale models; (4) \u201cbudgeted training,\u201d including some distinctive acceleration methods on source-constrained situations, e.g., for limitation on the total iterations; and (5) \u201csystem-centric,\u201d including some efficient distributed frameworks and open source libraries that provide adequate hardware support for the implementation of the above-mentioned acceleration algorithms. By presenting this comprehensive taxonomy, our survey presents a comprehensive review to understand the general mechanisms within each component and their joint interaction. Meanwhile, we further provide a detailed analysis and discussion of future works on the development of general acceleration techniques, which could inspire us to re-think and design novel efficient paradigms. Overall, we hope that this survey will serve as a valuable guideline for general efficient training.<\/jats:p>","DOI":"10.1145\/3700439","type":"journal-article","created":{"date-parts":[[2024,10,11]],"date-time":"2024-10-11T11:36:53Z","timestamp":1728646613000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":69,"title":["On Efficient Training of Large-Scale Deep Learning Models"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5659-3464","authenticated-orcid":false,"given":"Li","family":"Shen","sequence":"first","affiliation":[{"name":"Shenzhen Campus of Sun Yat-sen University, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2271-252X","authenticated-orcid":false,"given":"Yan","family":"Sun","sequence":"additional","affiliation":[{"name":"The University of Sydney, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3035-3839","authenticated-orcid":false,"given":"Zhiyuan","family":"Yu","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8976-2084","authenticated-orcid":false,"given":"Liang","family":"Ding","sequence":"additional","affiliation":[{"name":"JD.com Inc, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5952-8753","authenticated-orcid":false,"given":"Xinmei","family":"Tian","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7225-5449","authenticated-orcid":false,"given":"Dacheng","family":"Tao","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,11,11]]},"reference":[{"key":"e_1_3_1_2_2","article-title":"EZ-CLIP: Efficient zeroshot video action recognition","author":"Ahmad Shahzad","year":"2023","unstructured":"Shahzad Ahmad, Sukalpa Chanda, and Yogesh S. Rawat. 2023. EZ-CLIP: Efficient zeroshot video action recognition. arXiv preprint arXiv:2312.08010 (2023).","journal-title":"arXiv preprint arXiv:2312.08010"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5879544"},{"key":"e_1_3_1_4_2","first-page":"639","volume-title":"International Conference on Machine Learning","author":"Andriushchenko Maksym","year":"2022","unstructured":"Maksym Andriushchenko and Nicolas Flammarion. 2022. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning. PMLR, 639\u2013668."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3065737"},{"key":"e_1_3_1_6_2","article-title":"Greedy layer-wise training of deep networks","volume":"19","author":"Bengio Yoshua","year":"2006","unstructured":"Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. 2006. Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19 (2006), 153\u2013160.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553380"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2021.1977806"},{"key":"e_1_3_1_9_2","article-title":"Colossal-AI: A unified deep learning system for large-scale parallel training","author":"Bian Zhengda","year":"2021","unstructured":"Zhengda Bian, Hongxin Liu, Boxiang Wang, Haichen Huang, Yongbin Li, Chuanrui Wang, Fan Cui, and Yang You. 2021. Colossal-AI: A unified deep learning system for large-scale parallel training. arXiv preprint arXiv:2110.14883 (2021).","journal-title":"arXiv preprint arXiv:2110.14883"},{"key":"e_1_3_1_10_2","article-title":"Maximizing parallelism in distributed training for huge neural networks","author":"Bian Zhengda","year":"2021","unstructured":"Zhengda Bian, Qifan Xu, Boxiang Wang, and Yang You. 2021. Maximizing parallelism in distributed training for huge neural networks. arXiv preprint arXiv:2105.14450 (2021).","journal-title":"arXiv preprint arXiv:2105.14450"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.bigscience-1.9"},{"key":"e_1_3_1_12_2","article-title":"Token merging: Your ViT but faster","author":"Bolya Daniel","year":"2022","unstructured":"Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your ViT but faster. arXiv preprint arXiv:2210.09461 (2022).","journal-title":"arXiv preprint arXiv:2210.09461"},{"key":"e_1_3_1_13_2","article-title":"On the opportunities and risks of foundation models","author":"Bommasani Rishi","year":"2021","unstructured":"Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et\u00a0al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).","journal-title":"arXiv preprint arXiv:2108.07258"},{"key":"e_1_3_1_14_2","volume-title":"JAX: Composable Transformations of Python+NumPy Programs","author":"Bradbury James","year":"2018","unstructured":"James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: Composable Transformations of Python+NumPy Programs. http:\/\/github.com\/google\/jax"},{"key":"e_1_3_1_15_2","first-page":"1877","volume":"33","year":"2020","unstructured":"Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_16_2","article-title":"Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition","author":"Cai Han","year":"2022","unstructured":"Han Cai, Chuang Gan, and Song Han. 2022. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. arXiv preprint arXiv:2205.14756 (2022).","journal-title":"arXiv preprint arXiv:2205.14756"},{"key":"e_1_3_1_17_2","article-title":"Improving deep learning by inverse square root linear units (ISRLUs)","author":"Carlile Brad","year":"2017","unstructured":"Brad Carlile, Guy Delamarter, Paul Kinney, Akiko Marti, and Brian Whitney. 2017. Improving deep learning by inverse square root linear units (ISRLUs). arXiv preprint arXiv:1710.09967 (2017).","journal-title":"arXiv preprint arXiv:1710.09967"},{"key":"e_1_3_1_18_2","first-page":"64","article-title":"REX: Revisiting budgeted training with an improved schedule","volume":"4","author":"Chen John","year":"2022","unstructured":"John Chen, Cameron Wolfe, and Tasos Kyrillidis. 2022. REX: Revisiting budgeted training with an improved schedule. Proceedings of Machine Learning and Systems 4 (2022), 64\u201376.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01182"},{"key":"e_1_3_1_20_2","first-page":"20941","article-title":"Data-efficient GAN training beyond (just) augmentations: A lottery ticket perspective","volume":"34","author":"Chen Tianlong","year":"2021","unstructured":"Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. 2021. Data-efficient GAN training beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems 34 (2021), 20941\u201320955.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_21_2","first-page":"1597","volume-title":"International Conference on Machine Learning","author":"Chen Ting","year":"2020","unstructured":"Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning. PMLR, 1597\u20131607."},{"key":"e_1_3_1_22_2","article-title":"Symbolic discovery of optimization algorithms","author":"Chen Xiangning","year":"2023","unstructured":"Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et\u00a0al. 2023. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675 (2023).","journal-title":"arXiv preprint arXiv:2302.06675"},{"key":"e_1_3_1_23_2","article-title":"Mask-guided vision transformer (MG-ViT) for few-shot learning","author":"Chen Yuzhong","year":"2022","unstructured":"Yuzhong Chen, Zhenxiang Xiao, Lin Zhao, Lu Zhang, Haixing Dai, David Weizhong Liu, Zihao Wu, Changhe Li, Tuo Zhang, Changying Li, et\u00a0al. 2022. Mask-guided vision transformer (MG-ViT) for few-shot learning. arXiv preprint arXiv:2205.09995 (2022).","journal-title":"arXiv preprint arXiv:2205.09995"},{"key":"e_1_3_1_24_2","unstructured":"ZOMI Chen. 2024. AISys. https:\/\/github.com\/chenzomi12\/AISystem"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00135"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSI.2020.2964748"},{"key":"e_1_3_1_27_2","article-title":"Rethinking attention with performers","author":"Choromanski Krzysztof","year":"2020","unstructured":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et\u00a0al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 (2020).","journal-title":"arXiv preprint arXiv:2009.14794"},{"key":"e_1_3_1_28_2","article-title":"Fast and accurate deep network learning by exponential linear units (ELUs)","author":"Clevert Djork-Arn\u00e9","year":"2015","unstructured":"Djork-Arn\u00e9 Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 (2015).","journal-title":"arXiv preprint arXiv:1511.07289"},{"key":"e_1_3_1_29_2","article-title":"Autoaugment: Learning augmentation policies from data","author":"Cubuk Ekin D.","year":"2018","unstructured":"Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 (2018).","journal-title":"arXiv preprint arXiv:1805.09501"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00359"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01358"},{"key":"e_1_3_1_32_2","article-title":"ChatAug: Leveraging ChatGPT for text data augmentation","author":"Dai Haixing","year":"2023","unstructured":"Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, et\u00a0al. 2023. ChatAug: Leveraging ChatGPT for text data augmentation. arXiv preprint arXiv:2302.13007 (2023).","journal-title":"arXiv preprint arXiv:2302.13007"},{"key":"e_1_3_1_33_2","article-title":"Flashattention: Fast and memory-efficient exact attention with io-awareness","author":"Dao Tri","year":"2022","unstructured":"Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R\u00e9. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135 (2022).","journal-title":"arXiv preprint arXiv:2205.14135"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW55747.2022.00223"},{"key":"e_1_3_1_35_2","article-title":"Metainit: Initializing learning by learning to initialize","volume":"32","author":"Dauphin Yann N.","year":"2019","unstructured":"Yann N. Dauphin and Samuel Schoenholz. 2019. Metainit: Initializing learning by learning to initialize. Advances in Neural Information Processing Systems 32 (2019), 12624\u201312636.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1137\/1019005"},{"key":"e_1_3_1_37_2","article-title":"8-bit optimizers via block-wise quantization","author":"Dettmers Tim","year":"2021","unstructured":"Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861 (2021).","journal-title":"arXiv preprint arXiv:2110.02861"},{"key":"e_1_3_1_38_2","article-title":"Improved regularization of convolutional neural networks with cutout","author":"DeVries Terrance","year":"2017","unstructured":"Terrance DeVries and Graham W. Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).","journal-title":"arXiv preprint arXiv:1708.04552"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.3390\/info13070337"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.247"},{"key":"e_1_3_1_41_2","article-title":"Re-parameterizing your optimizers rather than architectures","author":"Ding Xiaohan","year":"2022","unstructured":"Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Kaiqi Huang, Jungong Han, and Guiguang Ding. 2022. Re-parameterizing your optimizers rather than architectures. arXiv preprint arXiv:2205.15242 (2022).","journal-title":"arXiv preprint arXiv:2205.15242"},{"issue":"7","key":"e_1_3_1_42_2","article-title":"Adaptive subgradient methods for online learning and stochastic optimization.","volume":"12","author":"Duchi John","year":"2011","unstructured":"John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12, 7 (2011), 2121\u20132159.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1201\/9781003214892","volume-title":"Uncertainty in Artificial Intelligence","author":"Dun Chen","year":"2022","unstructured":"Chen Dun, Cameron R. Wolfe, Christopher M. Jermaine, and Anastasios Kyrillidis. 2022. ResIST: Layer-wise decomposition of ResNets for distributed training. In Uncertainty in Artificial Intelligence. PMLR, 610\u2013620."},{"key":"e_1_3_1_44_2","article-title":"Understanding back-translation at scale","author":"Edunov Sergey","year":"2018","unstructured":"Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381 (2018).","journal-title":"arXiv preprint arXiv:1808.09381"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0277(93)90058-4"},{"key":"e_1_3_1_46_2","unstructured":"FairScale authors. 2021. FairScale: A General Purpose Modular PyTorch Library for High Performance and Large Scale Training. https:\/\/github.com\/facebookresearch\/fairscale"},{"key":"e_1_3_1_47_2","unstructured":"Vitaly Fedyunin. 2022. (Beta) Channels Last Memory Format in Pytorch. https:\/\/pytorch.org\/tutorials\/intermediate\/memory_format_tutorial.html#beta-channels-last-memory-format-in-pytorch"},{"key":"e_1_3_1_48_2","article-title":"A survey of data augmentation approaches for NLP","author":"Feng Steven Y.","year":"2021","unstructured":"Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. arXiv preprint arXiv:2105.03075 (2021).","journal-title":"arXiv preprint arXiv:2105.03075"},{"key":"e_1_3_1_49_2","article-title":"Sharpness-aware minimization for efficiently improving generalization","author":"Foret Pierre","year":"2020","unstructured":"Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020).","journal-title":"arXiv preprint arXiv:2010.01412"},{"key":"e_1_3_1_50_2","article-title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","author":"Frankle Jonathan","year":"2018","unstructured":"Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018).","journal-title":"arXiv preprint arXiv:1803.03635"},{"key":"e_1_3_1_51_2","first-page":"14873","article-title":"M-fac: Efficient matrix-free approximations of second-order information","volume":"34","author":"Frantar Elias","year":"2021","unstructured":"Elias Frantar, Eldar Kurtic, and Dan Alistarh. 2021. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems 34 (2021), 14873\u201314886.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_52_2","article-title":"Cramming: Training a language model on a single GPU in one day","author":"Geiping Jonas","year":"2022","unstructured":"Jonas Geiping and Tom Goldstein. 2022. Cramming: Training a language model on a single GPU in one day. arXiv preprint arXiv:2212.14034 (2022).","journal-title":"arXiv preprint arXiv:2212.14034"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics11060945"},{"key":"e_1_3_1_54_2","first-page":"249","volume-title":"Proceedings of the 13th International Conference on Artificial Intelligence and Statistics","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 249\u2013256."},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00162"},{"key":"e_1_3_1_56_2","first-page":"2337","volume-title":"International Conference on Machine Learning","author":"Gong Linyuan","year":"2019","unstructured":"Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Efficient training of bert by progressively stacking. In International Conference on Machine Learning. PMLR, 2337\u20132346."},{"key":"e_1_3_1_57_2","article-title":"Self-supervised pretraining of visual features in the wild","author":"Goyal Priya","year":"2021","unstructured":"Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et\u00a0al. 2021. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01988 (2021).","journal-title":"arXiv preprint arXiv:2103.01988"},{"key":"e_1_3_1_58_2","article-title":"On the transformer growth for progressive BERT training","author":"Gu Xiaotao","year":"2020","unstructured":"Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. 2020. On the transformer growth for progressive BERT training. arXiv preprint arXiv:2010.12562 (2020).","journal-title":"arXiv preprint arXiv:2010.12562"},{"key":"e_1_3_1_59_2","article-title":"Sequence-level mixed sample data augmentation","author":"Guo Demi","year":"2020","unstructured":"Demi Guo, Yoon Kim, and Alexander M. Rush. 2020. Sequence-level mixed sample data augmentation. arXiv preprint arXiv:2011.09039 (2020).","journal-title":"arXiv preprint arXiv:2011.09039"},{"key":"e_1_3_1_60_2","first-page":"1842","volume-title":"International Conference on Machine Learning","author":"Gupta Vineet","year":"2018","unstructured":"Vineet Gupta, Tomer Koren, and Yoram Singer. 2018. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning. PMLR, 1842\u20131850."},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1038\/35016072"},{"key":"e_1_3_1_62_2","article-title":"Turbo training with token dropout","author":"Han Tengda","year":"2022","unstructured":"Tengda Han, Weidi Xie, and Andrew Zisserman. 2022. Turbo training with token dropout. arXiv preprint arXiv:2210.04889 (2022).","journal-title":"arXiv preprint arXiv:2210.04889"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.08.002"},{"key":"e_1_3_1_64_2","article-title":"Accelerating parallel stochastic gradient descent via non-blocking mini-batches","author":"He Haoze","year":"2022","unstructured":"Haoze He and Parijat Dube. 2022. Accelerating parallel stochastic gradient descent via non-blocking mini-batches. arXiv preprint arXiv:2211.00889 (2022).","journal-title":"arXiv preprint arXiv:2211.00889"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01553"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00065"},{"key":"e_1_3_1_68_2","article-title":"Large-scale deep learning optimizations: A comprehensive survey","author":"He Xiaoxin","year":"2021","unstructured":"Xiaoxin He, Fuzhao Xue, Xiaozhe Ren, and Yang You. 2021. Large-scale deep learning optimizations: A comprehensive survey. arXiv preprint arXiv:2111.00856 (2021).","journal-title":"arXiv preprint arXiv:2111.00856"},{"key":"e_1_3_1_69_2","article-title":"Gaussian error linear units (GELUs)","author":"Hendrycks Dan","year":"2016","unstructured":"Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 (2016).","journal-title":"arXiv preprint arXiv:1606.08415"},{"key":"e_1_3_1_70_2","article-title":"Augmix: A simple data processing method to improve robustness and uncertainty","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2019. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 (2019).","journal-title":"arXiv preprint arXiv:1912.02781"},{"key":"e_1_3_1_71_2","article-title":"The forward-forward algorithm: Some preliminary investigations","author":"Hinton Geoffrey","year":"2022","unstructured":"Geoffrey Hinton. 2022. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345 (2022).","journal-title":"arXiv preprint arXiv:2212.13345"},{"key":"e_1_3_1_72_2","article-title":"Train longer, generalize better: Closing the generalization gap in large batch training of neural networks","volume":"30","author":"Hoffer Elad","year":"2017","unstructured":"Elad Hoffer, Itay Hubara, and Daniel Soudry. 2017. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. Advances in Neural Information Processing Systems 30 (2017), 1731\u20131741.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_73_2","article-title":"Training compute-optimal large language models","author":"Hoffmann Jordan","year":"2022","unstructured":"Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et\u00a0al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022).","journal-title":"arXiv preprint arXiv:2203.15556"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01462"},{"key":"e_1_3_1_75_2","unstructured":"Jeremy Howard. 2018. Fastai - Progressive Resizing. https:\/\/www.fast.ai\/2018\/04\/30\/dawnbench-fastai\/"},{"key":"e_1_3_1_76_2","first-page":"2098","volume-title":"International Conference on Machine Learning","author":"Huo Zhouyuan","year":"2018","unstructured":"Zhouyuan Huo, Bin Gu, Heng Huang, et\u00a0al. 2018. Decoupled parallel backpropagation with convergence guarantee. In International Conference on Machine Learning. PMLR, 2098\u20132106."},{"key":"e_1_3_1_77_2","article-title":"Averaging weights leads to wider optima and better generalization","author":"Izmailov Pavel","year":"2018","unstructured":"Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018).","journal-title":"arXiv preprint arXiv:1803.05407"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.831"},{"key":"e_1_3_1_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00507"},{"key":"e_1_3_1_80_2","first-page":"9895","article-title":"Sparse is enough in scaling transformers","volume":"34","author":"Jaszczur Sebastian","year":"2021","unstructured":"Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva. 2021. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems 34 (2021), 9895\u20139907.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD55607.2022.00068"},{"key":"e_1_3_1_82_2","article-title":"Accelerating deep learning by focusing on the biggest losers","author":"Jiang Angela H.","year":"2019","unstructured":"Angela H. Jiang, Daniel L.-K. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C. Lipton, et\u00a0al. 2019. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762 (2019).","journal-title":"arXiv preprint arXiv:1910.00762"},{"key":"e_1_3_1_83_2","volume-title":"Has It Trained Yet? NeurIPS 2022 Workshop","author":"Kaddour Jean","year":"2022","unstructured":"Jean Kaddour. 2022. Stop wasting my time! Saving days of ImageNet and BERT training with latest weight averaging. In Has It Trained Yet? NeurIPS 2022 Workshop. https:\/\/openreview.net\/forum?id=0OrABUHZuz"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-10925-7_24"},{"key":"e_1_3_1_85_2","article-title":"Scaling laws for neural language models","author":"Kaplan Jared","year":"2020","unstructured":"Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020).","journal-title":"arXiv preprint arXiv:2001.08361"},{"key":"e_1_3_1_86_2","article-title":"Progressive growing of GANs for improved quality, stability, and variation","author":"Karras Tero","year":"2017","unstructured":"Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).","journal-title":"arXiv preprint arXiv:1710.10196"},{"key":"e_1_3_1_87_2","article-title":"Biased importance sampling for deep neural network training","author":"Katharopoulos Angelos","year":"2017","unstructured":"Angelos Katharopoulos and Fran\u00e7ois Fleuret. 2017. Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043 (2017).","journal-title":"arXiv preprint arXiv:1706.00043"},{"key":"e_1_3_1_88_2","first-page":"2525","volume-title":"International Conference on Machine Learning","author":"Katharopoulos Angelos","year":"2018","unstructured":"Angelos Katharopoulos and Fran\u00e7ois Fleuret. 2018. Not all samples are created equal: Deep learning with importance sampling. In International Conference on Machine Learning. PMLR, 2525\u20132534."},{"key":"e_1_3_1_89_2","first-page":"669","volume-title":"International Conference on Artificial Intelligence and Statistics","author":"Kawaguchi Kenji","year":"2020","unstructured":"Kenji Kawaguchi and Haihao Lu. 2020. Ordered SGD: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics. PMLR, 669\u2013679."},{"key":"e_1_3_1_90_2","first-page":"18661","article-title":"Supervised contrastive learning","volume":"33","author":"Khosla Prannay","year":"2020","unstructured":"Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in Neural Information Processing Systems 33 (2020), 18661\u201318673.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_91_2","article-title":"Co-mixup: Saliency guided joint mixup with supermodular diversity","author":"Kim Jang-Hyun","year":"2021","unstructured":"Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. 2021. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2102.03065 (2021).","journal-title":"arXiv preprint arXiv:2102.03065"},{"key":"e_1_3_1_92_2","article-title":"Adam: A method for stochastic optimization","author":"Kingma Diederik P.","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"e_1_3_1_93_2","article-title":"Reformer: The efficient transformer","author":"Kitaev Nikita","year":"2020","unstructured":"Nikita Kitaev, \u0141ukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020).","journal-title":"arXiv preprint arXiv:2001.04451"},{"key":"e_1_3_1_94_2","article-title":"Self-normalizing neural networks","volume":"30","author":"Klambauer G\u00fcnter","year":"2017","unstructured":"G\u00fcnter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. Advances in Neural Information Processing Systems 30 (2017), 971\u2013980.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV56688.2023.00561"},{"key":"e_1_3_1_96_2","doi-asserted-by":"crossref","unstructured":"Adam Kohan Edward A. Rietman and Hava T. Siegelmann. 2024. Signal propagation: The framework for learning and inference in a forward pass. IEEE Transactions on Neural Networks and Learning Systems 35 (2024) 8585\u20138596.","DOI":"10.1109\/TNNLS.2022.3230914"},{"key":"e_1_3_1_97_2","article-title":"Error forward-propagation: Reusing feedforward connections to propagate errors in deep learning","author":"Kohan Adam A.","year":"2018","unstructured":"Adam A. Kohan, Edward A. Rietman, and Hava T. Siegelmann. 2018. Error forward-propagation: Reusing feedforward connections to propagate errors in deep learning. arXiv preprint arXiv:1808.03357 (2018).","journal-title":"arXiv preprint arXiv:1808.03357"},{"key":"e_1_3_1_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"e_1_3_1_99_2","article-title":"On weight initialization in deep neural networks","author":"Kumar Siddharth Krishna","year":"2017","unstructured":"Siddharth Krishna Kumar. 2017. On weight initialization in deep neural networks. arXiv preprint arXiv:1704.08863 (2017).","journal-title":"arXiv preprint arXiv:1704.08863"},{"key":"e_1_3_1_100_2","doi-asserted-by":"crossref","unstructured":"Guillaume Leclerc Andrew Ilyas Logan Engstrom Sung Min Park Hadi Salman and Aleksander Ma\u0327dry. 2023. FFCV: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 12011\u201312020.","DOI":"10.1109\/CVPR52729.2023.01156"},{"key":"e_1_3_1_101_2","article-title":"Stability and generalization for minibatch SGD and local SGD","author":"Lei Yunwen","year":"2023","unstructured":"Yunwen Lei, Tao Sun, and Mingrui Liu. 2023. Stability and generalization for minibatch SGD and local SGD. arXiv preprint arXiv:2310.01139 (2023).","journal-title":"arXiv preprint arXiv:2310.01139"},{"key":"e_1_3_1_102_2","first-page":"5809","volume-title":"International Conference on Machine Learning","author":"Lei Yunwen","year":"2020","unstructured":"Yunwen Lei and Yiming Ying. 2020. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning. PMLR, 5809\u20135819."},{"key":"e_1_3_1_103_2","article-title":"Shallow-to-deep training for neural machine translation","author":"Li Bei","year":"2020","unstructured":"Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu. 2020. Shallow-to-deep training for neural machine translation. arXiv preprint arXiv:2010.03737 (2020).","journal-title":"arXiv preprint arXiv:2010.03737"},{"key":"e_1_3_1_104_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01216"},{"key":"e_1_3_1_105_2","volume-title":"International Conference on Learning Representations","author":"Li Mengtian","year":"2019","unstructured":"Mengtian Li, Ersin Yumer, and Deva Ramanan. 2019. Budgeted training: Rethinking deep neural network training under resource constraints. In International Conference on Learning Representations."},{"key":"e_1_3_1_106_2","article-title":"Sequence parallelism: Making 4d parallelism possible","author":"Li Shenggui","year":"2021","unstructured":"Shenggui Li, Fuzhao Xue, Yongbin Li, and Yang You. 2021. Sequence parallelism: Making 4d parallelism possible. arXiv preprint arXiv:2105.13120 (2021).","journal-title":"arXiv preprint arXiv:2105.13120"},{"key":"e_1_3_1_107_2","article-title":"Trainable weight averaging for fast convergence and better generalization","author":"Li Tao","year":"2022","unstructured":"Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. 2022. Trainable weight averaging for fast convergence and better generalization. arXiv preprint arXiv:2205.13104 (2022).","journal-title":"arXiv preprint arXiv:2205.13104"},{"key":"e_1_3_1_108_2","article-title":"Deep model fusion: A survey","author":"Li Weishi","year":"2023","unstructured":"Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. 2023. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698 (2023).","journal-title":"arXiv preprint arXiv:2309.15698"},{"key":"e_1_3_1_109_2","doi-asserted-by":"publisher","DOI":"10.1137\/15M1007173"},{"key":"e_1_3_1_110_2","first-page":"5958","volume-title":"International Conference on Machine Learning","author":"Li Zhuohan","year":"2020","unstructured":"Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning. PMLR, 5958\u20135968."},{"key":"e_1_3_1_111_2","unstructured":"Xingyu Liao Peng Cheng Tianhe Ren Depeng Liang Kai Dang Yi Wang and Xiaoyu Xu. 2021. LiBai. https:\/\/github.com\/Oneflow-Inc\/libai"},{"key":"e_1_3_1_112_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41583-020-0277-3"},{"key":"e_1_3_1_113_2","article-title":"Don\u2019t use large mini-batches, use local SGD","author":"Lin Tao","year":"2018","unstructured":"Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. 2018. Don\u2019t use large mini-batches, use local SGD. arXiv preprint arXiv:1808.07217 (2018).","journal-title":"arXiv preprint arXiv:1808.07217"},{"key":"e_1_3_1_114_2","doi-asserted-by":"crossref","unstructured":"Tianyang Lin Yuxin Wang Xiangyang Liu and Xipeng Qiu. 2022. A survey of transformers. AI Open 3 (2022) 111\u2013132.","DOI":"10.1016\/j.aiopen.2022.10.001"},{"key":"e_1_3_1_115_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19809-0_26"},{"key":"e_1_3_1_116_2","first-page":"13925","volume-title":"International Conference on Machine Learning","author":"Liu Rui","year":"2022","unstructured":"Rui Liu and Barzan Mozafari. 2022. Communication-efficient distributed learning for large batch optimization. In International Conference on Machine Learning. PMLR, 13925\u201313946."},{"key":"e_1_3_1_117_2","article-title":"Roberta: A robustly optimized bert pretraining approach","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"e_1_3_1_118_2","article-title":"SGDR: Stochastic gradient descent with warm restarts","author":"Loshchilov Ilya","year":"2016","unstructured":"Ilya Loshchilov and Frank Hutter. 2016. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016).","journal-title":"arXiv preprint arXiv:1608.03983"},{"key":"e_1_3_1_119_2","article-title":"Decoupled weight decay regularization","author":"Loshchilov Ilya","year":"2017","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).","journal-title":"arXiv preprint arXiv:1711.05101"},{"key":"e_1_3_1_120_2","unstructured":"Sindy L\u00f6we Peter O\u2019Connor and Bastiaan Veeling. 2019. Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in Neural Information Processing Systems 32 (2019) 3033\u20133045."},{"key":"e_1_3_1_121_2","article-title":"Maximizing communication efficiency for large-scale training via 0\/1 Adam","author":"Lu Yucheng","year":"2022","unstructured":"Yucheng Lu, Conglong Li, Minjia Zhang, Christopher De Sa, and Yuxiong He. 2022. Maximizing communication efficiency for large-scale training via 0\/1 Adam. arXiv preprint arXiv:2202.06009 (2022).","journal-title":"arXiv preprint arXiv:2202.06009"},{"key":"e_1_3_1_122_2","unstructured":"Haoyu Ma Chengming Zhang Lizhi Xiang Xiaolong Ma Geng Yuan Wenkai Zhang Shiwei Liu Tianlong Chen Dingwen Tao Yanzhi Wang Zhangyang Wang and Xiaohui Xie. 2023. HRBP: Hardware-friendly Regrouping Towards Block-wise Pruning for Sparse Training. https:\/\/openreview.net\/forum?id=OSS-yWzE9Yu"},{"issue":"1","key":"e_1_3_1_123_2","first-page":"105","article-title":"PaddlePaddle: An open-source deep learning platform from industrial practice","volume":"1","author":"Ma Yanjun","year":"2019","unstructured":"Yanjun Ma, Dianhai Yu, Tian Wu, and Haifeng Wang. 2019. PaddlePaddle: An open-source deep learning platform from industrial practice. Frontiers of Data and Domputing 1, 1 (2019), 105\u2013115.","journal-title":"Frontiers of Data and Domputing"},{"key":"e_1_3_1_124_2","first-page":"53038","article-title":"Fine-tuning language models with just forward passes","volume":"36","author":"Malladi Sadhika","year":"2023","unstructured":"Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems 36 (2023), 53038\u201353075.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_125_2","article-title":"Project CGX: Scalable deep learning on commodity GPUs","author":"Markov Ilia","year":"2021","unstructured":"Ilia Markov, Hamidreza Ramezani, and Dan Alistarh. 2021. Project CGX: Scalable deep learning on commodity GPUs. arXiv preprint arXiv:2111.08617 (2021).","journal-title":"arXiv preprint arXiv:2111.08617"},{"key":"e_1_3_1_126_2","first-page":"2408","volume-title":"International Conference on Machine Learning","author":"Martens James","year":"2015","unstructured":"James Martens and Roger Grosse. 2015. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning. PMLR, 2408\u20132417."},{"key":"e_1_3_1_127_2","article-title":"An empirical model of large-batch training","author":"McCandlish Sam","year":"2018","unstructured":"Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162 (2018).","journal-title":"arXiv preprint arXiv:1812.06162"},{"key":"e_1_3_1_128_2","first-page":"15360","volume-title":"International Conference on Machine Learning","author":"Meng Chenlin","year":"2022","unstructured":"Chenlin Meng, Linqi Zhou, Kristy Choi, Tri Dao, and Stefano Ermon. 2022. ButterflyFlow: Building invertible layers with butterfly matrices. In International Conference on Machine Learning. PMLR, 15360\u201315375."},{"key":"e_1_3_1_129_2","first-page":"15630","volume-title":"International Conference on Machine Learning","author":"Mindermann S\u00f6ren","year":"2022","unstructured":"S\u00f6ren Mindermann, Jan M. Brauner, Muhammed T. Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H\u00f6ltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, et\u00a0al. 2022. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning. PMLR, 15630\u201315649."},{"key":"e_1_3_1_130_2","article-title":"All you need is a good init","author":"Mishkin Dmytro","year":"2015","unstructured":"Dmytro Mishkin and Jiri Matas. 2015. All you need is a good init. arXiv preprint arXiv:1511.06422 (2015).","journal-title":"arXiv preprint arXiv:1511.06422"},{"key":"e_1_3_1_131_2","article-title":"Accelerating sparse deep neural networks","author":"Mishra Asit","year":"2021","unstructured":"Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. 2021. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 (2021).","journal-title":"arXiv preprint arXiv:2104.08378"},{"key":"e_1_3_1_132_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_3_1_133_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476209"},{"key":"e_1_3_1_134_2","article-title":"K-SAM: Sharpness-aware minimization at the speed of SGD","author":"Ni Renkun","year":"2022","unstructured":"Renkun Ni, Ping-yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. 2022. K-SAM: Sharpness-aware minimization at the speed of SGD. arXiv preprint arXiv:2210.12864 (2022).","journal-title":"arXiv preprint arXiv:2210.12864"},{"key":"e_1_3_1_135_2","unstructured":"NVIDIA. 2022. Apex. https:\/\/github.com\/NVIDIA\/apex"},{"key":"e_1_3_1_136_2","volume-title":"Machine Learning Systems: Design and Implementation","author":"Community Open Machine Learning Systems","year":"2022","unstructured":"Open Machine Learning Systems Community. 2022. Machine Learning Systems: Design and Implementation. GitHub. https:\/\/github.com\/openmlsys\/openmlsys-zh"},{"key":"e_1_3_1_137_2","article-title":"A review of neuroscience-inspired machine learning","author":"Ororbia Alexander","year":"2024","unstructured":"Alexander Ororbia, Ankur Mali, Adam Kohan, Beren Millidge, and Tommaso Salvatori. 2024. A review of neuroscience-inspired machine learning. arXiv preprint arXiv:2403.18929 (2024).","journal-title":"arXiv preprint arXiv:2403.18929"},{"key":"e_1_3_1_138_2","volume-title":"International Conference on Learning Representations","author":"Pan Rui","year":"2021","unstructured":"Rui Pan, Haishan Ye, and Tong Zhang. 2021. Eigencurve: Optimal learning rate schedule for SGD on quadratic objectives with skewed hessian spectrums. In International Conference on Learning Representations."},{"key":"e_1_3_1_139_2","volume-title":"The 11th International Conference on Learning Representations","author":"Pan Xuran","unstructured":"Xuran Pan, Xuan Jin, Yuan He, Shiji Song, Gao Huang, et\u00a0al. [n.d.]. Budgeted training for vision transformer. In The 11th International Conference on Learning Representations."},{"key":"e_1_3_1_140_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-25056-9_43"},{"key":"e_1_3_1_141_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksuci.2021.11.017"},{"key":"e_1_3_1_142_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.427"},{"key":"e_1_3_1_143_2","article-title":"SpeeChain: A speech toolkit for large-scale machine speech chain","author":"Qi Heli","year":"2023","unstructured":"Heli Qi, Sashi Novitasari, Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2023. SpeeChain: A speech toolkit for large-scale machine speech chain. arXiv preprint arXiv:2301.02966 (2023).","journal-title":"arXiv preprint arXiv:2301.02966"},{"key":"e_1_3_1_144_2","doi-asserted-by":"publisher","DOI":"10.1080\/10618600.2023.2204130"},{"key":"e_1_3_1_145_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11431-020-1647-3"},{"key":"e_1_3_1_146_2","article-title":"Self-attention does not need  \\(O (n^2)\\)  memory","author":"Rabe Markus N.","year":"2021","unstructured":"Markus N. Rabe and Charles Staats. 2021. Self-attention does not need \\(O (n^2)\\) memory. arXiv preprint arXiv:2112.05682 (2021).","journal-title":"arXiv preprint arXiv:2112.05682"},{"issue":"8","key":"e_1_3_1_147_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_1_148_2","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_3_1_149_2","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"e_1_3_1_150_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476205"},{"key":"e_1_3_1_151_2","article-title":"Warm: On the benefits of weight averaged reward models","author":"Ram\u00e9 Alexandre","year":"2024","unstructured":"Alexandre Ram\u00e9, Nino Vieillard, L\u00e9onard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187 (2024).","journal-title":"arXiv preprint arXiv:2401.12187"},{"key":"e_1_3_1_152_2","first-page":"14358","article-title":"Are all frames equal? Active sparse labeling for video action detection","volume":"35","author":"Rana Aayush","year":"2022","unstructured":"Aayush Rana and Yogesh Rawat. 2022. Are all frames equal? Active sparse labeling for video action detection. Advances in Neural Information Processing Systems 35 (2022), 14358\u201314373.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_153_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3406703"},{"key":"e_1_3_1_154_2","first-page":"551","volume-title":"USENIX Annual Technical Conference","author":"Ren Jie","year":"2021","unstructured":"Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO-offload: Democratizing billion-scale model training. In USENIX Annual Technical Conference. 551\u2013564."},{"key":"e_1_3_1_155_2","article-title":"In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning","author":"Rizve Mamshad Nayeem","year":"2021","unstructured":"Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. 2021. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329 (2021).","journal-title":"arXiv preprint arXiv:2101.06329"},{"key":"e_1_3_1_156_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560260"},{"key":"e_1_3_1_157_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSSC.2022.3141050"},{"key":"e_1_3_1_158_2","article-title":"Brain-inspired computational intelligence via predictive coding","author":"Salvatori Tommaso","year":"2023","unstructured":"Tommaso Salvatori, Ankur Mali, Christopher L. Buckley, Thomas Lukasiewicz, Rajesh P. N. Rao, Karl Friston, and Alexander Ororbia. 2023. Brain-inspired computational intelligence via predictive coding. arXiv preprint arXiv:2308.07870 (2023).","journal-title":"arXiv preprint arXiv:2308.07870"},{"key":"e_1_3_1_159_2","article-title":"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks","author":"Saxe Andrew M.","year":"2013","unstructured":"Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 (2013).","journal-title":"arXiv preprint arXiv:1312.6120"},{"key":"e_1_3_1_160_2","article-title":"What language model to train if you have one million GPU hours?","author":"Scao Teven Le","year":"2022","unstructured":"Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M. Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et\u00a0al. 2022. What language model to train if you have one million GPU hours? arXiv preprint arXiv:2210.15424 (2022).","journal-title":"arXiv preprint arXiv:2210.15424"},{"key":"e_1_3_1_161_2","first-page":"30365","volume-title":"International Conference on Machine Learning","author":"Schwarzer Max","year":"2023","unstructured":"Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. 2023. Bigger, better, faster: Human-level Atari with human-level efficiency. In International Conference on Machine Learning. PMLR, 30365\u201330380."},{"key":"e_1_3_1_162_2","article-title":"Nonlinear acceleration of stochastic algorithms","volume":"30","author":"Scieur Damien","year":"2017","unstructured":"Damien Scieur, Francis Bach, and Alexandre d\u2019Aspremont. 2017. Nonlinear acceleration of stochastic algorithms. Advances in Neural Information Processing Systems 30 (2017), 3982\u20133991.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_163_2","article-title":"Regularized nonlinear acceleration","volume":"29","author":"Scieur Damien","year":"2016","unstructured":"Damien Scieur, Alexandre d\u2019Aspremont, and Francis Bach. 2016. Regularized nonlinear acceleration. Advances In Neural Information Processing Systems 29 (2016), 712\u2013720.","journal-title":"Advances In Neural Information Processing Systems"},{"key":"e_1_3_1_164_2","doi-asserted-by":"crossref","unstructured":"Frank Seide Hao Fu Jasha Droppo Gang Li and Dong Yu. 2014. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In Interspeech ISCA 1058\u20131062.","DOI":"10.21437\/Interspeech.2014-274"},{"key":"e_1_3_1_165_2","article-title":"Horovod: Fast and easy distributed deep learning in TensorFlow","author":"Sergeev Alexander","year":"2018","unstructured":"Alexander Sergeev and Mike Del Balso. 2018. Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 (2018).","journal-title":"arXiv preprint arXiv:1802.05799"},{"key":"e_1_3_1_166_2","article-title":"GLU variants improve transformer","author":"Shazeer Noam","year":"2020","unstructured":"Noam Shazeer. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202 (2020).","journal-title":"arXiv preprint arXiv:2002.05202"},{"key":"e_1_3_1_167_2","article-title":"Megatron-lm: Training multi-billion parameter language models using model parallelism","author":"Shoeybi Mohammad","year":"2019","unstructured":"Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019).","journal-title":"arXiv preprint arXiv:1909.08053"},{"key":"e_1_3_1_168_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2017.58"},{"key":"e_1_3_1_169_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.515"},{"key":"e_1_3_1_170_2","doi-asserted-by":"publisher","DOI":"10.1117\/12.2520589"},{"key":"e_1_3_1_171_2","article-title":"Using Deepspeed and Megatron to train Megatron-Turing NLG 530b, a large-scale generative language model","author":"Smith Shaden","year":"2022","unstructured":"Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et\u00a0al. 2022. Using Deepspeed and Megatron to train Megatron-Turing NLG 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 (2022).","journal-title":"arXiv preprint arXiv:2201.11990"},{"key":"e_1_3_1_172_2","article-title":"Don\u2019t decay the learning rate, increase the batch size","author":"Smith Samuel L.","year":"2017","unstructured":"Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V. Le. 2017. Don\u2019t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489 (2017).","journal-title":"arXiv preprint arXiv:1711.00489"},{"key":"e_1_3_1_173_2","article-title":"DNN training acceleration via exploring GPGPU friendly sparsity","author":"Song Zhuoran","year":"2022","unstructured":"Zhuoran Song, Yihong Xu, Han Li, Naifeng Jing, Xiaoyao Liang, and Li Jiang. 2022. DNN training acceleration via exploring GPGPU friendly sparsity. arXiv preprint arXiv:2203.05705 (2022).","journal-title":"arXiv preprint arXiv:2203.05705"},{"key":"e_1_3_1_174_2","article-title":"How to train your VIT? Data, augmentation, and regularization in vision transformers","author":"Steiner Andreas","year":"2021","unstructured":"Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. 2021. How to train your VIT? Data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 (2021).","journal-title":"arXiv preprint arXiv:2106.10270"},{"key":"e_1_3_1_175_2","article-title":"Prioritizing samples in reinforcement learning with reducible loss","author":"Sujit Shivakanth","year":"2022","unstructured":"Shivakanth Sujit, Somjit Nath, Pedro H. M. Braga, and Samira Ebrahimi Kahou. 2022. Prioritizing samples in reinforcement learning with reducible loss. arXiv preprint arXiv:2208.10483 (2022).","journal-title":"arXiv preprint arXiv:2208.10483"},{"key":"e_1_3_1_176_2","article-title":"Ernie: Enhanced representation through knowledge integration","author":"Sun Yu","year":"2019","unstructured":"Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 (2019).","journal-title":"arXiv preprint arXiv:1904.09223"},{"key":"e_1_3_1_177_2","article-title":"Profiling and improving the PyTorch dataloader for high-latency storage: A technical report","author":"Svogor Ivan","year":"2022","unstructured":"Ivan Svogor, Christian Eichenberger, Markus Spanring, Moritz Neun, and Michael Kopp. 2022. Profiling and improving the PyTorch dataloader for high-latency storage: A technical report. arXiv preprint arXiv:2211.04908 (2022).","journal-title":"arXiv preprint arXiv:2211.04908"},{"key":"e_1_3_1_178_2","article-title":"Convolutional neural networks with low-rank regularization","author":"Tai Cheng","year":"2015","unstructured":"Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et\u00a0al. 2015. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067 (2015).","journal-title":"arXiv preprint arXiv:1511.06067"},{"key":"e_1_3_1_179_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics11142183"},{"key":"e_1_3_1_180_2","first-page":"10096","volume-title":"International Conference on Machine Learning","author":"Tan Mingxing","year":"2021","unstructured":"Mingxing Tan and Quoc Le. 2021. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning. PMLR, 10096\u201310106."},{"key":"e_1_3_1_181_2","first-page":"10118","volume-title":"International Conference on Machine Learning","author":"Tang Hanlin","year":"2021","unstructured":"Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 2021. 1-bit Adam: Communication efficient large-scale training with Adam\u2019s convergence speed. In International Conference on Machine Learning. PMLR, 10118\u201310129."},{"key":"e_1_3_1_182_2","doi-asserted-by":"publisher","DOI":"10.1145\/3530811"},{"key":"e_1_3_1_183_2","article-title":"Diffusion with forward models: Solving stochastic inverse problems without direct supervision","volume":"36","author":"Tewari Ayush","year":"2024","unstructured":"Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Fr\u00e9do Durand, Bill Freeman, and Vincent Sitzmann. 2024. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Advances in Neural Information Processing Systems 36 (2024), 12349\u201312362.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_184_2","unstructured":"The Mosaic ML Team. 2021. Composer. https:\/\/github.com\/mosaicml\/composer\/"},{"issue":"2","key":"e_1_3_1_185_2","first-page":"26","article-title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","volume":"4","author":"Tieleman Tijmen","year":"2012","unstructured":"Tijmen Tieleman, Geoffrey Hinton, et\u00a0al. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning 4, 2 (2012), 26\u201331.","journal-title":"COURSERA: Neural Networks for Machine Learning"},{"key":"e_1_3_1_186_2","article-title":"Fixing the train-test resolution discrepancy","volume":"32","author":"Touvron Hugo","year":"2019","unstructured":"Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herv\u00e9 J\u00e9gou. 2019. Fixing the train-test resolution discrepancy. Advances in Neural Information Processing Systems 32 (2019), 8250\u20138260.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_187_2","article-title":"Efficient methods for natural language processing: A survey","author":"Treviso Marcos","year":"2022","unstructured":"Marcos Treviso, Tianchu Ji, Ji-Ung Lee, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Pedro H. Martins, et\u00a0al. 2022. Efficient methods for natural language processing: A survey. arXiv preprint arXiv:2209.00099 (2022).","journal-title":"arXiv preprint arXiv:2209.00099"},{"key":"e_1_3_1_188_2","article-title":"Saliencymix: A saliency guided data augmentation strategy for better regularization","author":"Uddin AFM","year":"2020","unstructured":"AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et\u00a0al. 2020. Saliencymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.01791 (2020).","journal-title":"arXiv preprint arXiv:2006.01791"},{"key":"e_1_3_1_189_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 5998\u20136008.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_190_2","first-page":"6438","volume-title":"International Conference on Machine Learning","author":"Verma Vikas","year":"2019","unstructured":"Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning. PMLR, 6438\u20136447."},{"key":"e_1_3_1_191_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i14.29477"},{"key":"e_1_3_1_192_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00881"},{"key":"e_1_3_1_193_2","article-title":"Linformer: Self-attention with linear complexity","author":"Wang Sinong","year":"2020","unstructured":"Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020).","journal-title":"arXiv preprint arXiv:2006.04768"},{"key":"e_1_3_1_194_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1306"},{"key":"e_1_3_1_195_2","article-title":"EfficientTrain: Exploring generalized curriculum learning for training visual backbones","author":"Wang Yulin","year":"2022","unstructured":"Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. 2022. EfficientTrain: Exploring generalized curriculum learning for training visual backbones. arXiv preprint arXiv:2211.09703 (2022).","journal-title":"arXiv preprint arXiv:2211.09703"},{"key":"e_1_3_1_196_2","article-title":"Emergent abilities of large language models","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et\u00a0al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).","journal-title":"arXiv preprint arXiv:2206.07682"},{"key":"e_1_3_1_197_2","article-title":"EDA: Easy data augmentation techniques for boosting performance on text classification tasks","author":"Wei Jason","year":"2019","unstructured":"Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196 (2019).","journal-title":"arXiv preprint arXiv:1901.11196"},{"key":"e_1_3_1_198_2","article-title":"The transformer family version 2.0","author":"Weng Lilian","year":"2023","unstructured":"Lilian Weng. 2023. The transformer family version 2.0. lilianweng.github.io (Jan. 2023). https:\/\/lilianweng.github.io\/posts\/2023-01-27-the-transformer-family-v2\/","journal-title":"lilianweng.github.io"},{"key":"e_1_3_1_199_2","article-title":"Fastformer: Additive attention can be all you need","author":"Wu Chuhan","year":"2021","unstructured":"Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2021. Fastformer: Additive attention can be all you need. arXiv preprint arXiv:2108.09084 (2021).","journal-title":"arXiv preprint arXiv:2108.09084"},{"key":"e_1_3_1_200_2","volume-title":"International Conference on Learning Representations","author":"Wu Xiaoxia","year":"2021","unstructured":"Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. 2021. When do curricula work? In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=tW4QEInpni"},{"key":"e_1_3_1_201_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i3.20181"},{"key":"e_1_3_1_202_2","volume-title":"Advances in Neural Information Processing Systems","author":"Xie Jian","year":"2022","unstructured":"Jian Xie, Jingwei Xu, Guochang Wang, Yuan Yao, Zenan Li, Chun Cao, and Hanghang Tong. 2022. A deep learning dataloader with shared data preparation. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_1_203_2","article-title":"Data selection for language models via importance resampling","author":"Xie Sang Michael","year":"2023","unstructured":"Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. 2023. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169 (2023).","journal-title":"arXiv preprint arXiv:2302.03169"},{"key":"e_1_3_1_204_2","doi-asserted-by":"publisher","unstructured":"Xingyu Xie Pan Zhou Huan Li Zhouchen Lin and Shuicheng Yan. 2022. Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models. DOI:10.48550\/ARXIV.2208.06677","DOI":"10.48550\/ARXIV.2208.06677"},{"key":"e_1_3_1_205_2","first-page":"38801","volume-title":"International Conference on Machine Learning","author":"Xu Hang","year":"2023","unstructured":"Hang Xu, Wenxuan Zhang, Jiawei Fei, Yuzhe Wu, TingWen Xie, Jun Huang, Yuchen Xie, Mohamed Elhoseiny, and Panos Kalnis. 2023. Slamb: Accelerated large batch training with sparse communication. In International Conference on Machine Learning. PMLR, 38801\u201338825."},{"key":"e_1_3_1_206_2","article-title":"An efficient 2d method for training super-large deep learning models","author":"Xu Qifan","year":"2021","unstructured":"Qifan Xu, Shenggui Li, Chaoyu Gong, and Yang You. 2021. An efficient 2d method for training super-large deep learning models. arXiv preprint arXiv:2104.05343 (2021).","journal-title":"arXiv preprint arXiv:2104.05343"},{"key":"e_1_3_1_207_2","article-title":"Deepspeed-chat: Easy, fast and affordable RLHF training of ChatGPT-like models at all scales","author":"Yao Zhewei","year":"2023","unstructured":"Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et\u00a0al. 2023. Deepspeed-chat: Easy, fast and affordable RLHF training of ChatGPT-like models at all scales. arXiv preprint arXiv:2308.01320 (2023).","journal-title":"arXiv preprint arXiv:2308.01320"},{"key":"e_1_3_1_208_2","article-title":"Large batch training of convolutional networks","author":"You Yang","year":"2017","unstructured":"Yang You, Igor Gitman, and Boris Ginsburg. 2017. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 (2017).","journal-title":"arXiv preprint arXiv:1708.03888"},{"key":"e_1_3_1_209_2","doi-asserted-by":"publisher","DOI":"10.1145\/3225058.3225069"},{"key":"e_1_3_1_210_2","article-title":"Decentralized training of foundation models in heterogeneous environments","author":"Yuan Binhang","year":"2022","unstructured":"Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. 2022. Decentralized training of foundation models in heterogeneous environments. arXiv preprint arXiv:2206.01288 (2022).","journal-title":"arXiv preprint arXiv:2206.01288"},{"key":"e_1_3_1_211_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00612"},{"key":"e_1_3_1_212_2","article-title":"Adadelta: An adaptive learning rate method","author":"Zeiler Matthew D.","year":"2012","unstructured":"Matthew D. Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012).","journal-title":"arXiv preprint arXiv:1212.5701"},{"key":"e_1_3_1_213_2","article-title":"Root mean square layer normalization","volume":"32","author":"Zhang Biao","year":"2019","unstructured":"Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems 32 (2019), 12360\u201312371.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_214_2","article-title":"mixup: Beyond empirical risk minimization","author":"Zhang Hongyi","year":"2017","unstructured":"Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017).","journal-title":"arXiv preprint arXiv:1710.09412"},{"key":"e_1_3_1_215_2","article-title":"Autoassist: A framework to accelerate training of deep neural networks","volume":"32","author":"Zhang Jiong","year":"2019","unstructured":"Jiong Zhang, Hsiang-Fu Yu, and Inderjit S. Dhillon. 2019. Autoassist: A framework to accelerate training of deep neural networks. Advances in Neural Information Processing Systems 32 (2019), 5996\u20136006.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_216_2","article-title":"Deep learning with elastic averaging SGD","volume":"28","author":"Zhang Sixin","year":"2015","unstructured":"Sixin Zhang, Anna E. Choromanska, and Yann LeCun. 2015. Deep learning with elastic averaging SGD. Advances in Neural Information Processing Systems 28 (2015), 685\u2013693.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_217_2","article-title":"Character-level convolutional networks for text classification","volume":"28","author":"Zhang Xiang","year":"2015","unstructured":"Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems 28 (2015), 649\u2013657.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_218_2","article-title":"ZerO initialization: Initializing neural networks with only zeros and ones","author":"Zhao Jiawei","year":"2021","unstructured":"Jiawei Zhao, Florian Sch\u00e4fer, and Anima Anandkumar. 2021. ZerO initialization: Initializing neural networks with only zeros and ones. arXiv preprint arXiv:2110.12661 (2021).","journal-title":"arXiv preprint arXiv:2110.12661"},{"key":"e_1_3_1_219_2","first-page":"7559","article-title":"Differentiable augmentation for data-efficient GAN training","volume":"33","author":"Zhao Shengyu","year":"2020","unstructured":"Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. 2020. Differentiable augmentation for data-efficient GAN training. Advances in Neural Information Processing Systems 33 (2020), 7559\u20137570.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_220_2","doi-asserted-by":"publisher","DOI":"10.3390\/a11100159"},{"key":"e_1_3_1_221_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.7000"},{"key":"e_1_3_1_222_2","article-title":"A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT","author":"Zhou Ce","year":"2023","unstructured":"Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et\u00a0al. 2023. A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT. arXiv preprint arXiv:2302.09419 (2023).","journal-title":"arXiv preprint arXiv:2302.09419"},{"key":"e_1_3_1_223_2","article-title":"A survey on efficient training of transformers","author":"Zhuang Bohan","year":"2023","unstructured":"Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, and Chunhua Shen. 2023. A survey on efficient training of transformers. arXiv preprint arXiv:2302.01107 (2023).","journal-title":"arXiv preprint arXiv:2302.01107"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3700439","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3700439","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:38Z","timestamp":1750295858000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3700439"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,11]]},"references-count":222,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3700439"],"URL":"https:\/\/doi.org\/10.1145\/3700439","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,11]]},"assertion":[{"value":"2023-06-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-04","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}