{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T16:01:06Z","timestamp":1774540866812,"version":"3.50.1"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,2]]},"abstract":"<jats:p>\n            Transformer-based large language models (LLMs) have demonstrated outstanding performance across diverse domains, particularly in the emerging\n            <jats:italic toggle=\"yes\">pretrain-then-finetune<\/jats:italic>\n            paradigm. LoRA, a parameter-efficient fine-tuning method, is commonly used to adapt a base LLM to multiple downstream tasks. Further, LLM platforms enable developers to fine-tune multiple models and develop various domain-specific applications simultaneously. However, existing model parallelism schemes suffer from high communication overhead and inefficient GPU utilization.\n          <\/jats:p>\n          <jats:p>In this paper, we present mLoRA, a parallelism-efficient fine-tuning system designed for training multiple LoRA across GPUs and machines. mLoRA introduces a novel LoRA-aware pipeline parallelism scheme that efficiently pipelines LoRA adapters and their distinct fine-tuning stages across GPUs and machines, along with a new LoRA-efficient operator to enhance GPU utilization. Our extensive evaluation shows that mLoRA can significantly reduce average fine-tuning task completion time, e.g., by 30%, compared to state-of-the-art methods like FSDP. More importantly, mLoRA enables simultaneous fine-tuning of larger models, e.g., two Llama-2-13B models on four NVIDIA RTX A6000 48GB GPUs, which is not feasible for FSDP due to high memory requirements. Hence, mLoRA not only increases fine-tuning efficiency but also makes it more accessible on cost-effective GPUs.<\/jats:p>","DOI":"10.14778\/3725688.3725718","type":"journal-article","created":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T14:19:21Z","timestamp":1756477161000},"page":"1948-1961","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs"],"prefix":"10.14778","volume":"18","author":[{"given":"Zhengmao","family":"Ye","sequence":"first","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dengchun","family":"Li","sequence":"additional","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zetao","family":"Hu","sequence":"additional","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tingfeng","family":"Lan","sequence":"additional","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jian","family":"Sha","sequence":"additional","affiliation":[{"name":"Ant Group"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shicong","family":"Zhang","sequence":"additional","affiliation":[{"name":"Zhejiang New Internet Exchange Center"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Duan","sequence":"additional","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Zuo","sequence":"additional","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"Lu","sequence":"additional","affiliation":[{"name":"The University of Texas at Arlington"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuanchun","family":"Zhou","sequence":"additional","affiliation":[{"name":"CNIC, Chinese Academy of Science"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingjie","family":"Tang","sequence":"additional","affiliation":[{"name":"Sichuan University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,8,29]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2023. Alpaca-LoRA. https:\/\/github.com\/tloen\/alpaca-lora."},{"key":"e_1_2_1_2_1","unstructured":"2024. Announcing Anyscale Private Endpoints and Anyscale Endpoints Finetuning. https:\/\/www.anyscale.com\/."},{"key":"e_1_2_1_3_1","unstructured":"2024. Customize a model with azure open AI service. https:\/\/learn.microsoft.com\/en-us\/azure\/ai-services\/openai\/how-to\/fine-tuning."},{"key":"e_1_2_1_4_1","unstructured":"2024. Hugging Face. https:\/\/huggingface.co\/."},{"key":"e_1_2_1_5_1","unstructured":"2024. NVIDIA Nsight Systems. https:\/\/developer.nvidia.com\/nsight-systems."},{"key":"e_1_2_1_6_1","unstructured":"2024. OpenAI fine-tuning. https:\/\/platform.openai.com\/docs\/guides\/fine-tuning."},{"key":"e_1_2_1_7_1","unstructured":"2024. SciPy: Solve a nonlinear least-squares problem with bounds on the variables. https:\/\/docs.scipy.org\/doc\/scipy-1.13.0\/reference\/generated\/scipy.optimize.least_squares.html."},{"key":"e_1_2_1_8_1","volume-title":"Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, and Arash Bakhtiari et al.","author":"Abdin Marah","year":"2024","unstructured":"Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, and Arash Bakhtiari et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL]"},{"key":"e_1_2_1_9_1","volume-title":"Anna Korhonen, and Ivan Vuli\u0107.","author":"Ansell Alan","year":"2021","unstructured":"Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vuli\u0107. 2021. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.07560 (2021)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.14778\/3626292.3626294"},{"key":"e_1_2_1_11_1","volume-title":"ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. arXiv preprint arXiv:2205.11961","author":"Asai Akari","year":"2022","unstructured":"Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. arXiv preprint arXiv:2205.11961 (2022)."},{"key":"e_1_2_1_12_1","volume-title":"vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training. arXiv preprint arXiv:2312.12391","author":"Bang Jehyeon","year":"2023","unstructured":"Jehyeon Bang, Yujeong Choi, Myeongwoo Kim, Yongdeok Kim, and Minsoo Rhu. 2023. vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training. arXiv preprint arXiv:2312.12391 (2023)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.3390\/rs16091477"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3320060"},{"key":"e_1_2_1_15_1","unstructured":"Rishi Bommasani Drew A Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)."},{"key":"e_1_2_1_16_1","volume-title":"Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839","author":"Chen Chi-Chung","year":"2018","unstructured":"Chi-Chung Chen, Chia-Lin Yang, and Hsiang-Yun Cheng. 2018. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839 (2018)."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2310.18547"},{"key":"e_1_2_1_18_1","volume-title":"Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174","author":"Chen Tianqi","year":"2016","unstructured":"Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Xie Chen Adam Eversole Gang Li Dong Yu and Frank Seide. 2012. Pipelined Back-Propagation for Context-Dependent Deep Neural Networks.. In Interspeech. 26\u201329.","DOI":"10.21437\/Interspeech.2012-7"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2020.3048836"},{"key":"e_1_2_1_21_1","volume-title":"Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168","author":"Cobbe Karl","year":"2021","unstructured":"Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168 (2021)."},{"key":"e_1_2_1_22_1","unstructured":"Tim Dettmers Mike Lewis Younes Belkada and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339 [cs.LG]"},{"key":"e_1_2_1_23_1","volume-title":"Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696","author":"Ding Ning","year":"2023","unstructured":"Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696 (2023)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-023-00626-4"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437801.3441593"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/3665844.3665857"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/3641204.3641221"},{"key":"e_1_2_1_28_1","unstructured":"Zeyu Han Chao Gao Jinyang Liu Jeff Zhang and Sai Qian Zhang. 2024. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey. arXiv:2403.14608 [cs.LG]"},{"key":"e_1_2_1_29_1","first-page":"211","article-title":"Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity","volume":"17","author":"Donglin Zhuang Zhongzhu Zhou Yuchao Li","year":"2024","unstructured":"Yuchao Li Donglin Zhuang Zhongzhu Zhou Xiafei Qiu Yong Li Wei Lin Shuaiwen Leon Song Haojun Xia, Zhen Zheng. 2024. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. Proceedings of the VLDB Endowment 17, 2 (2024), 211\u2013224.","journal-title":"Proceedings of the VLDB Endowment"},{"key":"e_1_2_1_30_1","volume-title":"International conference on machine learning. PMLR, 2790\u20132799","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. PMLR, 2790\u20132799."},{"key":"e_1_2_1_31_1","volume-title":"LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022","author":"Hu Edward J.","year":"2022","unstructured":"Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\u201329, 2022. OpenReview.net. https:\/\/openreview.net\/forum?id=nZeVKeeFYf9"},{"key":"e_1_2_1_32_1","volume-title":"Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32","author":"Huang Yanping","year":"2019","unstructured":"Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_1_33_1","first-page":"1","article-title":"Beyond Data and Model Parallelism for Deep Neural Networks","volume":"1","author":"Jia Zhihao","year":"2019","unstructured":"Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. Proceedings of Machine Learning and Systems 1 (2019), 1\u201313.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2310.01728"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of Machine Learning and Systems 5","author":"Korthikanti Vijay Anand","year":"2023","unstructured":"Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5 (2023)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3659437.3659449"},{"key":"e_1_2_1_38_1","volume-title":"The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691","author":"Lester Brian","year":"2021","unstructured":"Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)."},{"key":"e_1_2_1_39_1","volume-title":"MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts. arXiv preprint arXiv:2404.15159","author":"Li Dengchun","year":"2024","unstructured":"Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts. arXiv preprint arXiv:2404.15159 (2024)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","unstructured":"Ding Li and Zhang Xian. 2023. TianPeng: A Chinese chat model that is fine-tuned using LoRA on top of the LLaMA-30B model. https:\/\/huggingface.co\/pleisto\/tianpeng-lora-30B. 10.57967\/hf\/0528","DOI":"10.57967\/hf\/0528"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623612"},{"key":"e_1_2_1_42_1","first-page":"1725","article-title":"Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging","volume":"32","author":"Li Shigang","year":"2020","unstructured":"Shigang Li, Tal Ben-Nun, Giorgi Nadiradze, Salvatore Di Girolamo, Nikoli Dryden, Dan Alistarh, and Torsten Hoefler. 2020. Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging. IEEE Transactions on Parallel and Distributed Systems 32, 7 (2020), 1725\u20131739.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"e_1_2_1_43_1","volume-title":"PyTorch Distributed: Experiences on Accelerating Data Parallel Training. Proceedings of the VLDB Endowment 13","author":"Li Shen","unstructured":"Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. [n. d.]. PyTorch Distributed: Experiences on Accelerating Data Parallel Training. Proceedings of the VLDB Endowment 13, 12 ([n. d.])."},{"key":"e_1_2_1_44_1","volume-title":"Nam Sung Kim, and Alexander Schwing","author":"Li Youjie","year":"2018","unstructured":"Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. 2018. Pipe-SGD: A decentralized pipelined SGD framework for distributed deep net training. Advances in Neural Information Processing Systems 31 (2018)."},{"key":"e_1_2_1_45_1","volume-title":"International Conference on Machine Learning. PMLR, 3043\u20133052","author":"Lian Xiangru","year":"2018","unstructured":"Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. 2018. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning. PMLR, 3043\u20133052."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2402.09353"},{"key":"e_1_2_1_47_1","volume-title":"Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems 36","author":"Luo Gen","year":"2024","unstructured":"Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. 2024. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_2_1_48_1","volume-title":"PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https:\/\/github.com\/huggingface\/peft.","author":"Mangrulkar Sourab","year":"2022","unstructured":"Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https:\/\/github.com\/huggingface\/peft."},{"key":"e_1_2_1_49_1","volume-title":"Large language models: A survey. arXiv preprint arXiv:2402.06196","author":"Minaee Shervin","year":"2024","unstructured":"Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)."},{"key":"e_1_2_1_50_1","volume-title":"SwarmSGD: Scalable decentralized SGD with local updates. arXiv preprint arXiv:1910.12308","author":"Nadiradze Giorgi","year":"2019","unstructured":"Giorgi Nadiradze, Amirmojtaba Sabour, Dan Alistarh, Aditya Sharma, Ilia Markov, and Vitaly Aksenov. 2019. SwarmSGD: Scalable decentralized SGD with local updates. arXiv preprint arXiv:1910.12308 (2019)."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_2_1_52_1","volume-title":"International Conference on Machine Learning. PMLR, 7937\u20137947","author":"Narayanan Deepak","year":"2021","unstructured":"Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. 2021. Memory-efficient pipeline-parallel dnn training. In International Conference on Machine Learning. PMLR, 7937\u20137947."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476209"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2024.3352100"},{"key":"e_1_2_1_55_1","volume-title":"Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.3390\/app14052074"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3406703"},{"key":"e_1_2_1_59_1","volume-title":"Learning multiple visual domains with residual adapters. Advances in neural information processing systems 30","author":"Rebuffi Sylvestre-Alvise","year":"2017","unstructured":"Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_60_1","volume-title":"2021 USENIX Annual Technical Conference (USENIX ATC 21)","author":"Ren Jie","year":"2021","unstructured":"Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. {Zero-offload}: Democratizing {billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 551\u2013564."},{"key":"e_1_2_1_61_1","volume-title":"JetMoE: Reaching Llama2 Performance with 0.1 M Dollars. arXiv preprint arXiv:2404.07413","author":"Shen Yikang","year":"2024","unstructured":"Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. 2024. JetMoE: Reaching Llama2 Performance with 0.1 M Dollars. arXiv preprint arXiv:2404.07413 (2024)."},{"key":"e_1_2_1_62_1","volume-title":"S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285","author":"Sheng Ying","year":"2023","unstructured":"Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. 2023. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285 (2023)."},{"key":"e_1_2_1_63_1","volume-title":"Fairness in serving large language models. arXiv preprint arXiv:2401.00588","author":"Sheng Ying","year":"2023","unstructured":"Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E Gonzalez, and Ion Stoica. 2023. Fairness in serving large language models. arXiv preprint arXiv:2401.00588 (2023)."},{"key":"e_1_2_1_64_1","volume-title":"Juliette Love, et al.","author":"Team Gemma","year":"2024","unstructured":"Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi\u00e8re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)."},{"key":"e_1_2_1_65_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_2_1_66_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_2_1_67_1","first-page":"2440","article-title":"Hyperparameter optimization for Large Language Model instruction-tuning","volume":"711","author":"Tribes C","year":"2023","unstructured":"C Tribes, S Benarroch-Lelong, P Lu, and I Kobyzev. 2023. Hyperparameter optimization for Large Language Model instruction-tuning. Les Cahiers du GERAD ISSN 711 (2023), 2440.","journal-title":"Les Cahiers du GERAD ISSN"},{"key":"e_1_2_1_68_1","volume-title":"AI's New Frontier: Training Trillion-Parameter Models with Much Fewer GPUs. https:\/\/aibusiness.com\/nlp\/ai-s-new-frontier-training-trillion-parameter-models","author":"Wodecki Ben","year":"2024","unstructured":"Ben Wodecki. 2024. AI's New Frontier: Training Trillion-Parameter Models with Much Fewer GPUs. https:\/\/aibusiness.com\/nlp\/ai-s-new-frontier-training-trillion-parameter-models (2024)."},{"key":"e_1_2_1_69_1","unstructured":"Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu and Xin Jin. 2023. Fast Distributed Inference Serving for Large Language Models. arXiv:2305.05920 [cs.LG]"},{"key":"e_1_2_1_70_1","volume-title":"18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Wu Bingyang","year":"2024","unstructured":"Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 911\u2013927. https:\/\/www.usenix.org\/conference\/osdi24\/presentation\/wu-bingyang"},{"key":"e_1_2_1_71_1","unstructured":"Ming Xu. 2023. pycorrector: Text Error Correction Tool. https:\/\/github.com\/shibing624\/pycorrector"},{"key":"e_1_2_1_72_1","first-page":"269","article-title":"Pipemare: Asynchronous pipeline parallel dnn training","volume":"3","author":"Yang Bowen","year":"2021","unstructured":"Bowen Yang, Jian Zhang, Jonathan Li, Christopher R\u00e9, Christopher Aberger, and Christopher De Sa. 2021. Pipemare: Asynchronous pipeline parallel dnn training. Proceedings of Machine Learning and Systems 3 (2021), 269\u2013296.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_73_1","volume-title":"Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Yu Gyeong-In","year":"2022","unstructured":"Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 521\u2013538. https:\/\/www.usenix.org\/conference\/osdi22\/presentation\/yu"},{"key":"e_1_2_1_74_1","volume-title":"Understanding the overheads of launching CUDA kernels. ICPP19","author":"Zhang Lingqi","year":"2019","unstructured":"Lingqi Zhang, Mohamed Wahib, and Satoshi Matsuoka. 2019. Understanding the overheads of launching CUDA kernels. ICPP19 (2019), 5\u20138."},{"key":"e_1_2_1_75_1","unstructured":"Peiyuan Zhang Guangtao Zeng Tianduo Wang and Wei Lu. 2024. TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL]"},{"key":"e_1_2_1_76_1","volume-title":"The Eleventh International Conference on Learning Representations.","author":"Zhang Qingru","year":"2023","unstructured":"Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations."},{"key":"e_1_2_1_77_1","unstructured":"Justin Zhao Timothy Wang Wael Abid Geoffrey Angus Arnav Garg Jeffery Kinnison Alex Sherstinsky Piero Molino Travis Addair and Devvret Rishi. 2024. LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4 A Technical Report. arXiv preprint arXiv:2405.00732 (2024)."},{"key":"e_1_2_1_78_1","volume-title":"Masking as an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004.12406","author":"Zhao Mengjie","year":"2020","unstructured":"Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Sch\u00fctze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004.12406 (2020)."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611540.3611569"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3725688.3725718","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T14:23:29Z","timestamp":1756477409000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3725688.3725718"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2]]},"references-count":79,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,2]]}},"alternative-id":["10.14778\/3725688.3725718"],"URL":"https:\/\/doi.org\/10.14778\/3725688.3725718","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,2]]},"assertion":[{"value":"2025-08-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}