{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T18:22:17Z","timestamp":1789755737702,"version":"4.0.1"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,12,1]],"date-time":"2022-12-01T00:00:00Z","timestamp":1669852800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100016682","name":"VMware","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016682","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 42050105, 62072302, 62020106005, 62061146002, 61960206002"],"award-info":[{"award-number":["No. 42050105, 62072302, 62020106005, 62061146002, 61960206002"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000001","name":"US National Science Foundation","doi-asserted-by":"crossref","award":["NGSDI-2105564"],"award-info":[{"award-number":["NGSDI-2105564"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012247","name":"Program of Shanghai Academic\/Technology Research Leader","doi-asserted-by":"crossref","award":["No. 18XD1401800"],"award-info":[{"award-number":["No. 18XD1401800"]}],"id":[{"id":"10.13039\/501100012247","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:p>Training deep learning (DL) models in the cloud has become a norm. With the emergence of serverless computing and its benefits of true pay-as-you-go pricing and scalability, systems researchers have recently started to provide support for serverless-based training. However, the ability to train DL models on serverless platforms is hindered by the resource limitations of today's serverless infrastructure and DL models' explosive requirement for memory and bandwidth. This paper describes FuncPipe, a novel pipelined training framework specifically designed for serverless platforms that enable fast and low-cost training of DL models. FuncPipe is designed with the key insight that model partitioning can be leveraged to bridge both memory and bandwidth gaps between the capacity of serverless functions and the requirement of DL training. Conceptually simple, we have to answer several design questions, including how to partition the model, configure each serverless function, and exploit each function's uplink\/downlink bandwidth. In particular, we tailor a micro-batch scheduling policy for the serverless environment, which serves as the basis for the subsequent optimization. Our Mixed-Integer Quadratic Programming formulation automatically and simultaneously configures serverless resources and partitions models to fit within the resource constraints. Lastly, we improve the bandwidth efficiency of storage-based synchronization with a novel pipelined scatter-reduce algorithm. We implement FuncPipe on two popular cloud serverless platforms and show that it achieves 7%-77% cost savings and 1.3X-2.2X speedup compared to state-of-the-art serverless-based frameworks.<\/jats:p>","DOI":"10.1145\/3570607","type":"journal-article","created":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T20:20:10Z","timestamp":1670530810000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["FuncPipe: A Pipelined Serverless Framework for Fast and Cost-Efficient Training of Deep Learning Models"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0156-2454","authenticated-orcid":false,"given":"Yunzhuo","family":"Liu","sequence":"first","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6711-4342","authenticated-orcid":false,"given":"Bo","family":"Jiang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0060-2266","authenticated-orcid":false,"given":"Tian","family":"Guo","sequence":"additional","affiliation":[{"name":"Worcester Polytechnic Institute, Worcester, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5016-8472","authenticated-orcid":false,"given":"Zimeng","family":"Huang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5542-0342","authenticated-orcid":false,"given":"Wenhao","family":"Ma","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0357-8356","authenticated-orcid":false,"given":"Xinbing","family":"Wang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3331-2302","authenticated-orcid":false,"given":"Chenghu","family":"Zhou","sequence":"additional","affiliation":[{"name":"Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,8]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"SAND: Towards High-Performance Serverless Computing. In 2018 Usenix Annual Technical Conference (USENIX ATC 18)","author":"Akkus Istemi Ekin","year":"2018","unstructured":"Istemi Ekin Akkus , Ruichuan Chen , Ivica Rimac , Manuel Stein , Klaus Satzke , Andre Beck , Paarijaat Aditya , and Volker Hilt . 2018 . SAND: Towards High-Performance Serverless Computing. In 2018 Usenix Annual Technical Conference (USENIX ATC 18) . 923--935. Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein, Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt. 2018. SAND: Towards High-Performance Serverless Computing. In 2018 Usenix Annual Technical Conference (USENIX ATC 18). 923--935."},{"key":"e_1_2_1_2_1","unstructured":"Alibaba. 2022a. Alibaba Cloud Function Compute. https:\/\/www.aliyun.com\/product\/fc.  Alibaba. 2022a. Alibaba Cloud Function Compute. https:\/\/www.aliyun.com\/product\/fc."},{"key":"e_1_2_1_3_1","unstructured":"Alibaba. 2022b. Alibaba Cloud Function Compute Instance Type. https:\/\/help.aliyun.com\/document_detail\/179379.html.  Alibaba. 2022b. Alibaba Cloud Function Compute Instance Type. https:\/\/help.aliyun.com\/document_detail\/179379.html."},{"key":"e_1_2_1_4_1","unstructured":"Alibaba. 2022c. Alibaba Cloud Object Storage Service. https:\/\/www.aliyun.com\/product\/oss.  Alibaba. 2022c. Alibaba Cloud Object Storage Service. https:\/\/www.aliyun.com\/product\/oss."},{"key":"e_1_2_1_5_1","volume-title":"CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17)","author":"Alipourfard Omid","year":"2017","unstructured":"Omid Alipourfard , Hongqiang Harry Liu , Jianshu Chen , Shivaram Venkataraman , Minlan Yu , and Ming Zhang . 2017 . CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) . 469--482. Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. 2017. CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 469--482."},{"key":"e_1_2_1_6_1","unstructured":"Amazon. 2022. AWS Lambda. https:\/\/www.aliyun.com\/product\/fc.  Amazon. 2022. AWS Lambda. https:\/\/www.aliyun.com\/product\/fc."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-50743-5_5"},{"key":"e_1_2_1_8_1","volume-title":"Maximizing Parallelism in Distributed Training for Huge Neural Networks. arXiv preprint arXiv:2105.14450","author":"Bian Zhengda","year":"2021","unstructured":"Zhengda Bian , Qifan Xu , Boxiang Wang , and Yang You . 2021. Maximizing Parallelism in Distributed Training for Huge Neural Networks. arXiv preprint arXiv:2105.14450 ( 2021 ). Zhengda Bian, Qifan Xu, Boxiang Wang, and Yang You. 2021. Maximizing Parallelism in Distributed Training for Huge Neural Networks. arXiv preprint arXiv:2105.14450 (2021)."},{"key":"e_1_2_1_9_1","volume-title":"Advances in Neural Information Processing Systems","volume":"32","author":"Venkatakrishnan Shaileshh Bojja","year":"2019","unstructured":"Shaileshh Bojja Venkatakrishnan , Shreyan Gupta , Hongzi Mao , Mohammad Alizadeh , 2019 . Learning Generalizable Device Placement Algorithms for Distributed Machine Learning . Advances in Neural Information Processing Systems , Vol. 32 (2019). Shaileshh Bojja Venkatakrishnan, Shreyan Gupta, Hongzi Mao, Mohammad Alizadeh, et al. 2019. Learning Generalizable Device Placement Algorithms for Distributed Machine Learning. Advances in Neural Information Processing Systems , Vol. 32 (2019)."},{"key":"e_1_2_1_10_1","volume-title":"A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599","author":"Brochu Eric","year":"2010","unstructured":"Eric Brochu , Vlad M Cora , and Nando De Freitas . 2010. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 ( 2010 ). Eric Brochu, Vlad M Cora, and Nando De Freitas. 2010. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362711"},{"key":"e_1_2_1_12_1","volume-title":"Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839","author":"Chen Chi-Chung","year":"2018","unstructured":"Chi-Chung Chen , Chia-Lin Yang , and Hsiang-Yun Cheng . 2018. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839 ( 2018 ). Chi-Chung Chen, Chia-Lin Yang, and Hsiang-Yun Cheng. 2018. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839 (2018)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392717.3392771"},{"key":"e_1_2_1_14_1","volume-title":"Adabatch: Adaptive batch sizes for training deep neural networks. arXiv preprint arXiv:1712.02029","author":"Devarakonda Aditya","year":"2017","unstructured":"Aditya Devarakonda , Maxim Naumov , and Michael Garland . 2017 . Adabatch: Adaptive batch sizes for training deep neural networks. arXiv preprint arXiv:1712.02029 (2017). Aditya Devarakonda, Maxim Naumov, and Michael Garland. 2017. Adabatch: Adaptive batch sizes for training deep neural networks. arXiv preprint arXiv:1712.02029 (2017)."},{"key":"e_1_2_1_15_1","volume-title":"Jost Tobias Springenberg, and Frank Hutter","author":"Domhan Tobias","year":"2015","unstructured":"Tobias Domhan , Jost Tobias Springenberg, and Frank Hutter . 2015 . Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence. Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.3023302"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437801.3441593"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2018.00049"},{"key":"e_1_2_1_19_1","volume-title":"2019 USENIX Annual Technical Conference (USENIX ATC 19)","author":"Fouladi Sadjad","year":"2019","unstructured":"Sadjad Fouladi , Francisco Romero , Dan Iter , Qian Li , Shuvo Chatterjee , Christos Kozyrakis , Matei Zaharia , and Keith Winstein . 2019 . From laptop to lambda: Outsourcing everyday jobs to thousands of transient functional containers . In 2019 USENIX Annual Technical Conference (USENIX ATC 19) . 475--488. Sadjad Fouladi, Francisco Romero, Dan Iter, Qian Li, Shuvo Chatterjee, Christos Kozyrakis, Matei Zaharia, and Keith Winstein. 2019. From laptop to lambda: Outsourcing everyday jobs to thousands of transient functional containers. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 475--488."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3322795.3331461"},{"key":"e_1_2_1_21_1","first-page":"779","article-title":"Hydrozoa: Dynamic Hybrid-Parallel DNN Training on Serverless Containers","volume":"4","author":"Guo Runsheng","year":"2022","unstructured":"Runsheng Guo , Victor Guo , Antonio Kim , Josh Hildred , and Khuzaima Daudjee . 2022 . Hydrozoa: Dynamic Hybrid-Parallel DNN Training on Serverless Containers . Proceedings of Machine Learning and Systems , Vol. 4 (2022), 779 -- 794 . Runsheng Guo, Victor Guo, Antonio Kim, Josh Hildred, and Khuzaima Daudjee. 2022. Hydrozoa: Dynamic Hybrid-Parallel DNN Training on Serverless Containers. Proceedings of Machine Learning and Systems , Vol. 4 (2022), 779--794.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_22_1","unstructured":"Gurobi. 2022a. Gurobi - The Fastest Solver. https:\/\/www.gurobi.com.  Gurobi. 2022a. Gurobi - The Fastest Solver. https:\/\/www.gurobi.com."},{"key":"e_1_2_1_23_1","unstructured":"Gurobi. 2022b. Setting up and using a Floating license. https:\/\/www.gurobi.com\/documentation\/9.5\/quickstart_mac\/setting_up_and_using_a_flo.html.  Gurobi. 2022b. Setting up and using a Floating license. https:\/\/www.gurobi.com\/documentation\/9.5\/quickstart_mac\/setting_up_and_using_a_flo.html."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3464298.3476132"},{"key":"e_1_2_1_25_1","volume-title":"Serverless Computation with OpenLambda. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16)","author":"Hendrickson Scott","year":"2016","unstructured":"Scott Hendrickson , Stephen Sturdevant , Tyler Harter , and Venkataramani. 2016 . Serverless Computation with OpenLambda. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16) . Scott Hendrickson, Stephen Sturdevant, Tyler Harter, and Venkataramani. 2016. Serverless Computation with OpenLambda. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476223"},{"key":"e_1_2_1_27_1","volume-title":"Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems","author":"Huang Yanping","year":"2019","unstructured":"Yanping Huang , Youlong Cheng , Ankur Bapna , Orhan Firat , Dehao Chen , Mia Chen , HyoukJoong Lee , Jiquan Ngiam , Quoc V Le , Yonghui Wu , 2019 . Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems , Vol. 32 (2019). Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems , Vol. 32 (2019)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00049"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00049"},{"key":"e_1_2_1_30_1","volume-title":"AMPS-Inf: Automatic Model Partitioning for Serverless Inference with Cost Efficiency. In 50th International Conference on Parallel Processing. 1--12","author":"Jarachanthan Jananie","year":"2021","unstructured":"Jananie Jarachanthan , Li Chen , Fei Xu , and Bo Li . 2021 . AMPS-Inf: Automatic Model Partitioning for Serverless Inference with Cost Efficiency. In 50th International Conference on Parallel Processing. 1--12 . Jananie Jarachanthan, Li Chen, Fei Xu, and Bo Li. 2021. AMPS-Inf: Automatic Model Partitioning for Serverless Inference with Cost Efficiency. In 50th International Conference on Parallel Processing. 1--12."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419111.3421302"},{"key":"e_1_2_1_32_1","volume-title":"Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19)","author":"Jeon Myeongjae","year":"2019","unstructured":"Myeongjae Jeon , Shivaram Venkataraman , Amar Phanishayee , Junjie Qian , Wencong Xiao , and Fan Yang . 2019 . Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19) . 947--960. Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 947--960."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning","volume":"80","author":"Jia Zhihao","year":"2018","unstructured":"Zhihao Jia , Sina Lin , Charles R. Qi , and Alex Aiken . 2018 . Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks . In Proceedings of the 35th International Conference on Machine Learning , Vol. 80 . 2279--2288. Zhihao Jia, Sina Lin, Charles R. Qi, and Alex Aiken. 2018. Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80. 2279--2288."},{"key":"e_1_2_1_34_1","first-page":"1","article-title":"Beyond Data and Model Parallelism for Deep Neural Networks","volume":"1","author":"Jia Zhihao","year":"2019","unstructured":"Zhihao Jia , Matei Zaharia , and Alex Aiken . 2019 . Beyond Data and Model Parallelism for Deep Neural Networks . Proceedings of Machine Learning and Systems , Vol. 1 (2019), 1 -- 13 . Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. Proceedings of Machine Learning and Systems , Vol. 1 (2019), 1--13.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3459240"},{"key":"e_1_2_1_36_1","volume-title":"2018 USENIX Annual Technical Conference (USENIX ATC 18)","author":"Klimovic Ana","year":"2018","unstructured":"Ana Klimovic , Yawen Wang , Christos Kozyrakis , Patrick Stuedi , Jonas Pfefferle , and Animesh Trivedi . 2018 a. Understanding ephemeral storage for serverless analytics . In 2018 USENIX Annual Technical Conference (USENIX ATC 18) . 789--794. Ana Klimovic, Yawen Wang, Christos Kozyrakis, Patrick Stuedi, Jonas Pfefferle, and Animesh Trivedi. 2018a. Understanding ephemeral storage for serverless analytics. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 789--794."},{"key":"e_1_2_1_37_1","volume-title":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","author":"Klimovic Ana","year":"2018","unstructured":"Ana Klimovic , Yawen Wang , Patrick Stuedi , Animesh Trivedi , Jonas Pfefferle , and Christos Kozyrakis . 2018 b. Pocket: Elastic ephemeral storage for serverless analytics . In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . 427--444. Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. 2018b. Pocket: Elastic ephemeral storage for serverless analytics. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 427--444."},{"key":"e_1_2_1_38_1","first-page":"230","article-title":"A System for Massively Parallel Hyperparameter Tuning","volume":"2","author":"Li Liam","year":"2020","unstructured":"Liam Li , Kevin Jamieson , Afshin Rostamizadeh , Ekaterina Gonina , Jonathan Ben-tzur, Moritz Hardt , Benjamin Recht , and Ameet Talwalkar . 2020 a. A System for Massively Parallel Hyperparameter Tuning . In Proceedings of Machine Learning and Systems , Vol. 2. 230 -- 246 . Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2020a. A System for Massively Parallel Hyperparameter Tuning. In Proceedings of Machine Learning and Systems, Vol. 2. 230--246.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476145"},{"key":"e_1_2_1_40_1","volume-title":"Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning. In 2021 IEEE 41th International Conference on Distributed Computing Systems (ICDCS).","author":"Li Shijian","year":"2021","unstructured":"Shijian Li , Oren Mangoubi , Lijie Xu , and Tian Guo . 2021 . Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning. In 2021 IEEE 41th International Conference on Distributed Computing Systems (ICDCS). Shijian Li, Oren Mangoubi, Lijie Xu, and Tian Guo. 2021. Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning. In 2021 IEEE 41th International Conference on Distributed Computing Systems (ICDCS)."},{"key":"e_1_2_1_41_1","volume-title":"Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS).","author":"Li Shijian","year":"2020","unstructured":"Shijian Li , Robert J. Walls , and Tian Guo . 2020 b. Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). Shijian Li, Robert J. Walls, and Tian Guo. 2020b. Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415530"},{"key":"e_1_2_1_43_1","unstructured":"Ryan McDonald Keith Hall and Gideon Mann. 2010. Distributed training strategies for the structured perceptron. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics. 456--464.  Ryan McDonald Keith Hall and Gideon Mann. 2010. Distributed training strategies for the structured perceptron. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics. 456--464."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCSW.2017.36"},{"key":"e_1_2_1_45_1","unstructured":"Microsoft. 2022a. Microsoft Azure Cloud Computing. https:\/\/azure.microsoft.com\/.  Microsoft. 2022a. Microsoft Azure Cloud Computing. https:\/\/azure.microsoft.com\/."},{"key":"e_1_2_1_46_1","unstructured":"Microsoft. 2022b. Microsoft Azure Storage. https:\/\/azure.microsoft.com\/services\/storage.  Microsoft. 2022b. Microsoft Azure Storage. https:\/\/azure.microsoft.com\/services\/storage."},{"key":"e_1_2_1_47_1","unstructured":"Azalia Mirhoseini Anna Goldie Hieu Pham Benoit Steiner Quoc V. Le and Jeff Dean. 2018. Hierarchical Planning for Device Placement. In ICLR.  Azalia Mirhoseini Anna Goldie Hieu Pham Benoit Steiner Quoc V. Le and Jeff Dean. 2018. Hierarchical Planning for Device Placement. In ICLR."},{"key":"e_1_2_1_48_1","volume-title":"International Conference on Machine Learning. PMLR, 2430--2439","author":"Mirhoseini Azalia","year":"2017","unstructured":"Azalia Mirhoseini , Hieu Pham , Quoc V Le , Benoit Steiner , Rasmus Larsen , Yuefeng Zhou , Naveen Kumar , Mohammad Norouzi , Samy Bengio , and Jeff Dean . 2017 . Device placement optimization with reinforcement learning . In International Conference on Machine Learning. PMLR, 2430--2439 . Azalia Mirhoseini, Hieu Pham, Quoc V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. 2017. Device placement optimization with reinforcement learning. In International Conference on Machine Learning. PMLR, 2430--2439."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476209"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISAP.2005.1599245"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2007.370405"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2008.09.002"},{"key":"e_1_2_1_54_1","volume-title":"2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19)","author":"Rausch Thomas","year":"2019","unstructured":"Thomas Rausch , Waldemar Hummer , Vinod Muthusamy , Alexander Rashed , and Schahram Dustdar . 2019 . Towards a serverless platform for edge AI . In 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19) . Thomas Rausch, Waldemar Hummer, Vinod Muthusamy, Alexander Rashed, and Schahram Dustdar. 2019. Towards a serverless platform for edge AI. In 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19)."},{"key":"e_1_2_1_55_1","unstructured":"Baidu Research. 2017. baidu-allreduce. https:\/\/github.com\/baidu-research\/baidu-allreduce.  Baidu Research. 2017. baidu-allreduce. https:\/\/github.com\/baidu-research\/baidu-allreduce."},{"key":"e_1_2_1_56_1","volume-title":"Pragna Gopa, Paul Batum, Neeraja J Yadwadkar, Rodrigo Fonseca, Christos Kozyrakis, and Ricardo Bianchini.","author":"Romero Francisco","year":"2021","unstructured":"Francisco Romero , Gohar Irfan Chaudhry, \u00cd nigo Goiri , Pragna Gopa, Paul Batum, Neeraja J Yadwadkar, Rodrigo Fonseca, Christos Kozyrakis, and Ricardo Bianchini. 2021 a. Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications . (2021). Francisco Romero, Gohar Irfan Chaudhry, \u00cd nigo Goiri, Pragna Gopa, Paul Batum, Neeraja J Yadwadkar, Rodrigo Fonseca, Christos Kozyrakis, and Ricardo Bianchini. 2021a. Faa$T: A Transparent Auto-Scaling Cache for Serverless Applications. (2021)."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472883.3486972"},{"key":"e_1_2_1_58_1","first-page":"1","article-title":"Measuring the Effects of Data Parallelism on Neural Network Training","volume":"20","author":"Shallue Christopher J.","year":"2019","unstructured":"Christopher J. Shallue , Jaehoon Lee , Joseph Antognini , Jascha Sohl-Dickstein , Roy Frostig , and George E. Dahl . 2019 . Measuring the Effects of Data Parallelism on Neural Network Training . Journal of Machine Learning Research , Vol. 20 , 112 (2019), 1 -- 49 . Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. 2019. Measuring the Effects of Data Parallelism on Neural Network Training. Journal of Machine Learning Research , Vol. 20, 112 (2019), 1--49.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_59_1","volume-title":"Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems","author":"Shazeer Noam","year":"2018","unstructured":"Noam Shazeer , Youlong Cheng , Niki Parmar , Dustin Tran , Ashish Vaswani , Penporn Koanantakool , Peter Hawkins , 2018 . Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems , Vol. 31 (2018). Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, et al. 2018. Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems , Vol. 31 (2018)."},{"key":"e_1_2_1_60_1","volume-title":"Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053","author":"Shoeybi Mohammad","year":"2019","unstructured":"Mohammad Shoeybi , Mostofa Patwary , Raul Puri , Patrick LeGresley , Jared Casper , and Bryan Catanzaro . 2019 . Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)."},{"key":"e_1_2_1_61_1","volume-title":"Megan Leszczynski, Jian Zhang, and Christopher R\u00e9.","author":"Sohoni Nimit Sharad","year":"2019","unstructured":"Nimit Sharad Sohoni , Christopher Richard Aberger , Megan Leszczynski, Jian Zhang, and Christopher R\u00e9. 2019 . Low-memory neural network training: A technical report. arXiv preprint arXiv:1904.10631 (2019). Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszczynski, Jian Zhang, and Christopher R\u00e9. 2019. Low-memory neural network training: A technical report. arXiv preprint arXiv:1904.10631 (2019)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403342"},{"key":"e_1_2_1_63_1","first-page":"15451","article-title":"Efficient algorithms for device placement of dnn graph operators","volume":"33","author":"Tarnawski Jakub M","year":"2020","unstructured":"Jakub M Tarnawski , Amar Phanishayee , Nikhil Devanur , Divya Mahajan , and Fanny Nina Paravecino . 2020 . Efficient algorithms for device placement of dnn graph operators . Advances in Neural Information Processing Systems , Vol. 33 (2020), 15451 -- 15463 . Jakub M Tarnawski, Amar Phanishayee, Nikhil Devanur, Divya Mahajan, and Fanny Nina Paravecino. 2020. Efficient algorithms for device placement of dnn graph operators. Advances in Neural Information Processing Systems , Vol. 33 (2020), 15451--15463.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342005051521"},{"key":"e_1_2_1_65_1","volume-title":"15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)","author":"Thorpe John","year":"2021","unstructured":"John Thorpe , Yifan Qiao , Jonathan Eyolfson , Shen Teng , Guanzhou Hu , Zhihao Jia , Jinliang Wei , Keval Vora , Ravi Netravali , Miryung Kim , 2021 . Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads . In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21) . 495--514. John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, et al. 2021. Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). 495--514."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/COINS49042.2020.9191377"},{"key":"e_1_2_1_67_1","volume-title":"18th USENIX Conference on File and Storage Technologies (FAST 20)","author":"Wang Ao","year":"2020","unstructured":"Ao Wang , Jingyuan Zhang , Xiaolong Ma , Ali Anwar , Lukas Rupprecht , Dimitrios Skourtis , Vasily Tarasov , Feng Yan , and Yue Cheng . 2020 b. InfiniCache: Exploiting Ephemeral Serverless Functions to Build a Cost-Effective Memory Cache . In 18th USENIX Conference on File and Storage Technologies (FAST 20) . usenix.org, 267--281. Ao Wang, Jingyuan Zhang, Xiaolong Ma, Ali Anwar, Lukas Rupprecht, Dimitrios Skourtis, Vasily Tarasov, Feng Yan, and Yue Cheng. 2020b. InfiniCache: Exploiting Ephemeral Serverless Functions to Build a Cost-Effective Memory Cache. In 18th USENIX Conference on File and Storage Technologies (FAST 20). usenix.org, 267--281."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2019.8737391"},{"key":"e_1_2_1_69_1","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Wang Linnan","year":"2020","unstructured":"Linnan Wang , Rodrigo Fonseca , and Yuandong Tian . 2020 a. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search . In Advances in Neural Information Processing Systems (NeurIPS) , 2020 (online). Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. 2020a. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search. In Advances in Neural Information Processing Systems (NeurIPS), 2020 (online)."},{"key":"e_1_2_1_70_1","volume-title":"2018 USENIX Annual Technical Conference (USENIX ATC 18)","author":"Wang Liang","year":"2018","unstructured":"Liang Wang , Mengyuan Li , Yinqian Zhang , Thomas Ristenpart , and Michael Swift . 2018 . Peeking behind the curtains of serverless platforms . In 2018 USENIX Annual Technical Conference (USENIX ATC 18) . 133--146. Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, and Michael Swift. 2018. Peeking behind the curtains of serverless platforms. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 133--146."},{"key":"e_1_2_1_71_1","volume-title":"Boxer: Data Analytics on Network-enabled Serverless Platforms. In 11th Annual Conference on Innovative Data Systems Research.","author":"Wawrzoniak Ingo","year":"2021","unstructured":"Ingo Wawrzoniak , Mike and Fraga Barcelos Paulus Bruno . 2021 . Boxer: Data Analytics on Network-enabled Serverless Platforms. In 11th Annual Conference on Innovative Data Systems Research. Ingo Wawrzoniak, Mike and Fraga Barcelos Paulus Bruno. 2021. Boxer: Data Analytics on Network-enabled Serverless Platforms. In 11th Annual Conference on Innovative Data Systems Research."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2021.3054656"},{"key":"e_1_2_1_73_1","volume-title":"Gillis: Serving Large Neural Networks in Serverless Functions with Automatic Model Partitioning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS). IEEE, 138--148","author":"Yu Minchen","year":"2021","unstructured":"Minchen Yu , Zhifeng Jiang , 2021 . Gillis: Serving Large Neural Networks in Serverless Functions with Automatic Model Partitioning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS). IEEE, 138--148 . Minchen Yu, Zhifeng Jiang, et al. 2021. Gillis: Serving Large Neural Networks in Serverless Functions with Automatic Model Partitioning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS). IEEE, 138--148."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362723"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.5555\/3060832.3060950"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3405671.3405810"},{"key":"e_1_2_1_77_1","volume-title":"10th International Conference on Learning Representations, ICLR","author":"Zhao Yiyang","year":"2022","unstructured":"Yiyang Zhao , Linnan Wang , Kevin Yang , Tianjun Zhang , Tian Guo , and Yuandong Tian . 2022 . Multi-objective Optimization by Learning Space Partitions . 10th International Conference on Learning Representations, ICLR (2022). Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. 2022. Multi-objective Optimization by Learning Space Partitions. 10th International Conference on Learning Representations, ICLR (2022)."}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3570607","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3570607","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3570607","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:17Z","timestamp":1750178777000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3570607"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12]]},"references-count":77,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["10.1145\/3570607"],"URL":"https:\/\/doi.org\/10.1145\/3570607","relation":{},"ISSN":["2476-1249"],"issn-type":[{"value":"2476-1249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12]]},"assertion":[{"value":"2022-12-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}