{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T13:17:18Z","timestamp":1780492638118,"version":"3.54.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Manage. Inf. Syst."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>The surge in interest and application of large language models (LLMs) has sparked a drive to fine-tune these models to suit specific applications, such as finance and medical science. However, concerns regarding data privacy have emerged, especially when multiple stakeholders aim to collaboratively enhance LLMs using sensitive data. In this scenario, federated learning becomes a natural choice, allowing decentralized fine-tuning without exposing raw data to central servers. Motivated by this, we investigate how data privacy can be ensured in LLM fine-tuning through practical federated learning approaches, enabling secure contributions from multiple parties to enhance LLMs. Yet, challenges arise: (1) despite avoiding raw data exposure, there is a risk of inferring sensitive information from model outputs, and (2) federated learning for LLMs incurs notable communication overhead. To address these challenges, this article introduces DP-LoRA, a novel federated learning algorithm tailored for LLMs. DP-LoRA preserves data privacy by employing a Gaussian mechanism that adds noise in weight updates, maintaining individual data privacy while facilitating collaborative model training. Moreover, DP-LoRA optimizes communication efficiency via low-rank adaptation, minimizing the transmission of updated weights during distributed training. The experimental results across medical, financial, and general datasets using various LLMs demonstrate that DP-LoRA effectively ensures strict privacy constraints while minimizing communication overhead.<\/jats:p>","DOI":"10.1145\/3682068","type":"journal-article","created":{"date-parts":[[2024,8,13]],"date-time":"2024-08-13T11:10:46Z","timestamp":1723547446000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":44,"title":["Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9532-1709","authenticated-orcid":false,"given":"Xiao-Yang","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Computer Science, Rensselaer Polytechnic Institute; Department of Electrical Engineering, Columbia University, New York, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1601-5983","authenticated-orcid":false,"given":"Rongyi","family":"Zhu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Rochester, Rochester, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6677-7504","authenticated-orcid":false,"given":"Daochen","family":"Zha","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Rice University, Houston, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0628-1416","authenticated-orcid":false,"given":"Jiechao","family":"Gao","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Virginia, Charlottesville, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9479-6350","authenticated-orcid":false,"given":"Shan","family":"Zhong","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Columbia University, New York, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2546-8331","authenticated-orcid":false,"given":"Matt","family":"White","sequence":"additional","affiliation":[{"name":"Executive Director, PyTorch Foundation, GM of AI, Linux Foundation, California, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1004-0140","authenticated-orcid":false,"given":"Meikang","family":"Qiu","sequence":"additional","affiliation":[{"name":"School of Computer and Cyber Sciences, Augusta University, Augusta, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"e_1_3_4_2_2","unstructured":"Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray and others. 2022. Training language models to follow instructions with human feedback. In NeurIPS. 27730\u201327744."},{"key":"e_1_3_4_3_2","unstructured":"OpenAI. 2023. GPT-4 technical report. ArXiv abs\/2303.08774 2023."},{"key":"e_1_3_4_4_2","unstructured":"Shijie Wu Ozan Irsoy Steven Lu Vadim Dabravolski Mark Dredze Sebastian Gehrmann Prabhanjan Kambadur David Rosenberg and Gideon Mann. 2023. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303.17564 2023."},{"key":"e_1_3_4_5_2","doi-asserted-by":"crossref","unstructured":"Xiao-Yang Liu Guoxuan Wang Hongyang Yang and Daochen Zha. 2023. Fingpt: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485 (2023).","DOI":"10.2139\/ssrn.4489826"},{"key":"e_1_3_4_6_2","article-title":"FinGPT: Open-source financial large language models","author":"Yang Hongyang","year":"2023","unstructured":"Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. FinGPT: Open-source financial large language models. FinLLM at IJCAIarXiv preprint arXiv:2306.06031, (2023).","journal-title":"FinLLM at IJCAI"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","unstructured":"Xiao-Yang Liu Ziyi Xia Hongyang Yang Jiechao Gao Daochen Zha Ming Zhu Christina Dan Wang Zhaoran Wang and Jian Guo. 2024. Dynamic datasets and market environments for financial reinforcement learning. Mach. Learn. 113 5 (2024) 2795\u20132839. DOI:10.1007\/S10994-023-06511-W","DOI":"10.1007\/S10994-023-06511-W"},{"key":"e_1_3_4_8_2","unstructured":"Qianqian Xie Weiguang Han Xiao Zhang Yanzhao Lai Min Peng Alejandro Lopez-Lira and Jimin Huang. 2023. PIXIU: A large language model instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443 2023."},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2978575"},{"key":"e_1_3_4_10_2","unstructured":"Karan Singhal Shekoofeh Azizi Tao Tu S. Sara Mahdavi Jason Wei Hyung Won Chung Nathan Scales Ajay Tanwani Heather Cole-Lewis Stephen Pfohl and others. 2023. Large language models encode clinical knowledge. Nature (2023) 1\u20139."},{"key":"e_1_3_4_11_2","unstructured":"Ha-Thanh Nguyen. 2023. A brief report on LawGPT 1.0: A virtual legal assistant based on GPT-3. arXiv preprint arXiv:2302.05729 2023."},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbac409"},{"key":"e_1_3_4_13_2","unstructured":"Daochen Zha Zaid Pervaiz Bhat Kwei-Herng Lai Fan Yang Zhimeng Jiang Shaochen Zhong and Xia Hu. 2023. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158 2023."},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611977653.ch106"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3589335.3641297"},{"key":"e_1_3_4_16_2","unstructured":"Qinggang Zhang Junnan Dong Hao Chen Xiao Huang Daochen Zha and Zailiang Yu. 2023. Knowgpt: Black-box knowledge injection for large language models. arXiv preprint arXiv:2312.06185 2023."},{"key":"e_1_3_4_17_2","article-title":"LoRA: Low-rank adaptation of large language models","author":"Hu Edward J.","year":"2021","unstructured":"Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (2021).","journal-title":"In Proceedings of the International Conference on Learning Representations"},{"key":"e_1_3_4_18_2","unstructured":"Jesse Dodge Gabriel Ilharco Roy Schwartz Ali Farhadi Hannaneh Hajishirzi and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations data orders and early stopping. arXiv preprint arXiv:2002.06305."},{"key":"e_1_3_4_19_2","article-title":"Differentially private fine-tuning of language models","author":"Yu Da","year":"2021","unstructured":"Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, and Lukas Wutschitz. 2021. Differentially private fine-tuning of language models. In Proceedings of the International Conference on Learning Representations (2021).","journal-title":"In Proceedings of the International Conference on Learning Representations"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-023-00626-4"},{"key":"e_1_3_4_21_2","unstructured":"Yansong Li Zhixing Tan and Yang Liu. 2023. Privacy-preserving prompt tuning for large language model services. arXiv preprint arXiv:2305.06212. 2023."},{"key":"e_1_3_4_22_2","article-title":"Privacy and data protection in ChatGPT and other AI chatbots: Strategies for securing user information","author":"Sebastian Glorin","year":"2023","unstructured":"Glorin Sebastian. 2023. Privacy and data protection in ChatGPT and other AI chatbots: Strategies for securing user information. Available at SSRN 4454761 (2023).","journal-title":"Available at SSRN 4454761"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2020.106854"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.106775"},{"key":"e_1_3_4_25_2","first-page":"2633","volume-title":"Proceedings of the 30th USENIX Security Symposium","author":"Carlini Nicholas","year":"2021","unstructured":"Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium. 2633\u20132650."},{"key":"e_1_3_4_26_2","unstructured":"Siwon Kim Sangdoo Yun Hwaran Lee Martin Gubri Sungroh Yoon and Seong Joon Oh. 2023. Propile: Probing privacy leakage in large language models. arXiv preprint arXiv:2307.01881."},{"key":"e_1_3_4_27_2","unstructured":"Haoran Li Dadi Guo Wei Fan Mingshi Xu and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197."},{"key":"e_1_3_4_28_2","unstructured":"Jakub Kone\u010dn\u1ef3 H. Brendan McMahan Felix X. Yu Peter Richt\u00e1rik Ananda Theertha Suresh and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492."},{"key":"e_1_3_4_29_2","first-page":"3973","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Hamer Jenny","year":"2020","unstructured":"Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. 2020. Fedboost: A communication-efficient algorithm for federated learning. In Proceedings of the International Conference on Machine Learning. PMLR, 3973\u20133983."},{"key":"e_1_3_4_30_2","doi-asserted-by":"crossref","unstructured":"Guangchen Lan Xiao-Yang Liu Yijing Zhang and Xiaodong Wang. 2023. Communication-efficient federated learning for resource-constrained edge devices. IEEE Transactions on Machine Learning in Communications and Networking 1 (2023) 210\u2013224.","DOI":"10.1109\/TMLCN.2023.3309773"},{"key":"e_1_3_4_31_2","first-page":"267","volume-title":"Proceedings of the 28th USENIX Security Symposium","author":"Carlini Nicholas","year":"2019","unstructured":"Nicholas Carlini, Chang Liu, \u00dalfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Security Symposium. 267\u2013284."},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1201\/9781351006620"},{"key":"e_1_3_4_33_2","first-page":"29","volume-title":"Proceedings of the IEEE\/WIC\/ACM International Conference on Web Intelligence","author":"Qiu MeiKang","year":"2004","unstructured":"MeiKang Qiu, Kang Zhang, and Maolin Huang. 2004. An empirical study of web interface design on small display devices. In Proceedings of the IEEE\/WIC\/ACM International Conference on Web Intelligence. IEEE, 29\u201335."},{"key":"e_1_3_4_34_2","unstructured":"Weiyan Shi Aiqi Cui Evan Li Ruoxi Jia and Zhou Yu. 2021. Selective differential privacy for language modeling. arXiv preprint arXiv:2108.12944."},{"key":"e_1_3_4_35_2","doi-asserted-by":"crossref","unstructured":"Rohan Anil Badih Ghazi Vineet Gupta Ravi Kumar and Pasin Manurangsi. 2021. Large-scale differentially private BERT. arXiv preprint arXiv:2108.01624.","DOI":"10.18653\/v1\/2022.findings-emnlp.484"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.102"},{"key":"e_1_3_4_37_2","unstructured":"Xuechen Li Florian Tramer Percy Liang and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679 2021."},{"key":"e_1_3_4_38_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong Yifan Du Chen Yang Yushuo Chen Zhipeng Chen Jinhao Jiang Ruiyang Ren Yifan Li Xinyu Tang Zikang Liu Peiyu Liu Jian-Yun Nie and Ji-Rong Wen. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 2023."},{"issue":"8","key":"e_1_3_4_39_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_4_40_2","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D. Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell and others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. 1877\u20131901."},{"key":"e_1_3_4_41_2","doi-asserted-by":"crossref","unstructured":"Sid Black Stella Biderman Eric Hallahan Quentin Anthony Leo Gao Laurence Golding Horace He Connor Leahy Kyle McDonell Jason Phang Michael Pieler USVSN Sai Prashanth Shivanshu Purohit Laria Reynolds Jonathan Tow Ben Wang and Samuel Weinbach. 2022. GPT-NEOX-20B: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 2022.","DOI":"10.18653\/v1\/2022.bigscience-1.9"},{"key":"e_1_3_4_42_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018."},{"key":"e_1_3_4_43_2","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann Parker Schuh Kensen Shi Sasha Tsvyashchenko Joshua Maynez Abhishek Rao Parker Barnes Yi Tay Noam Shazeer Vinodkumar Prabhakaran Emily Reif Nan Du Ben Hutchinson Reiner Pope James Bradbury Jacob Austin Michael Isard Guy Gur-Ari Pengcheng Yin Toju Duke Anselm Levskaya Sanjay Ghemawat Sunipa Dev Henryk Michalewski Xavier Garcia Vedant Misra Kevin Robinson Liam Fedus Denny Zhou Daphne Ippolito David Luan Hyeontaek Lim Barret Zoph Alexander Spiridonov Ryan Sepassi David Dohan Shivani Agrawal Mark Omernick Andrew M. Dai Thanumalayan Sankaranarayana Pillai Marie Pellat Aitor Lewkowycz Erica Moreira Rewon Child Oleksandr Polozov Katherine Lee Zongwei Zhou Xuezhi Wang Brennan Saeta Mark Diaz Orhan Firat Michele Catasta Jason Wei Kathy Meier-Hellstern Douglas Eck Jeff Dean Slav Petrov and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022."},{"key":"e_1_3_4_44_2","unstructured":"Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ili\u0107 Daniel Hesslow Roman Castagn\u00e9 Alexandra Sasha Luccioni Fran\u00e7ois Yvon Matthias Gall\u00e9 Jonathan Tow Alexander M. Rush Stella Biderman Albert Webson Pawan Sasanka Ammanamanchi Thomas Wang Beno\u00eet Sagot Niklas Muennighoff Albert Villanova del Moral Olatunji Ruwase Rachel Bawden Stas Bekman Angelina McMillan-Major Iz Beltagy Huu Nguyen Lucile Saulnier Samson Tan Pedro Ortiz Suarez Victor Sanh Hugo Lauren\u00e7on Yacine Jernite Julien Launay Margaret Mitchell Colin Raffel Aaron Gokaslan Adi Simhi Aitor Soroa Alham Fikri Aji Amit Alfassy Anna Rogers Ariel Kreisberg Nitzav Canwen Xu Chenghao Mou Chris Emezue Christopher Klamm Colin Leong Daniel van Strien David Ifeoluwa Adelani Dragomir Radev Eduardo Gonz\u00e1lez Ponferrada Efrat Levkovizh Ethan Kim Eyal Bar Natan Francesco De Toni G\u00e9rard Dupont Germ\u00e1n Kruszewski Giada Pistilli Hady Elsahar Hamza Benyamina Hieu Tran Ian Yu Idris Abdulmumin Isaac Johnson Itziar Gonzalez-Dios Javier de la Rosa Jenny Chim Jesse Dodge Jian Zhu Jonathan Chang J\u00f6rg Frohberg Joseph Tobing Joydeep Bhattacharjee Khalid Almubarak Kimbo Chen Kyle Lo Leandro Von Werra Leon Weber Long Phan Loubna Ben allal Ludovic Tanguy Manan Dey Manuel Romero Mu\u00f1oz Maraim Masoud Mar\u00eda Grandury Mario \u0160a\u0161ko Max Huang Maximin Coavoux Mayank Singh Mike Tian-Jian Jiang Minh Chien Vu Mohammad A. Jauhar Mustafa Ghaleb Nishant Subramani Nora Kassner Nurulaqilla Khamis Olivier Nguyen Omar Espejel Ona de Gibert Paulo Villegas Peter Henderson Pierre Colombo Priscilla Amuok Quentin Lhoest Rheza Harliman Rishi Bommasani Roberto Luis L\u00f3pez Rui Ribeiro Salomey Osei Sampo Pyysalo Sebastian Nagel Shamik Bose Shamsuddeen Hassan Muhammad Shanya Sharma Shayne Longpre Somaieh Nikpoor Stanislav Silberberg Suhas Pai Sydney Zink Tiago Timponi Torrent Timo Schick Tristan Thrush Valentin Danchev Vassilina Nikoulina Veronika Laippala Violette Lepercq Vrinda Prabhu Zaid Alyafeai Zeerak Talat Arun Raja Benjamin Heinzerling Chenglei Si Davut Emre Ta\u015far Elizabeth Salesky Sabrina J. Mielke Wilson Y. Lee Abheesht Sharma Andrea Santilli Antoine Chaffin Arnaud Stiegler Debajyoti Datta Eliza Szczechla Gunjan Chhablani Han Wang Harshit Pandey Hendrik Strobelt Jason Alan Fries Jos Rozen Leo Gao Lintang Sutawika M Saiful Bari Maged S. Al-shaibani Matteo Manica Nihal Nayak Ryan Teehan Samuel Albanie Sheng Shen Srulik Ben-David Stephen H. Bach Taewoon Kim Tali Bers Thibault Fevry Trishala Neeraj Urmish Thakker Vikas Raunak Xiangru Tang Zheng-Xin Yong Zhiqing Sun Shaked Brody Yallow Uri Hadar Tojarieh Adam Roberts Hyung Won Chung Jaesung Tae Jason Phang Ofir Press Conglong Li Deepak Narayanan Hatim Bourfoune Jared Casper Jeff Rasley Max Ryabinin Mayank Mishra Minjia Zhang Mohammad Shoeybi Myriam Peyrounette Nicolas Patry Nouamane Tazi Omar Sanseviero Patrick von Platen Pierre Cornette Pierre Fran\u00e7ois Lavall\u00e9e R\u00e9mi Lacroix Samyam Rajbhandari Sanchit Gandhi Shaden Smith St\u00e9phane Requena Suraj Patil Tim Dettmers Ahmed Baruwa Amanpreet Singh Anastasia Cheveleva Anne-Laure Ligozat Arjun Subramonian Aur\u00e9lie N\u00e9v\u00e9ol Charles Lovering Dan Garrette Deepak Tunuguntla Ehud Reiter Ekaterina Taktasheva Ekaterina Voloshina Eli Bogdanov Genta Indra Winata Hailey Schoelkopf Jan-Christoph Kalo Jekaterina Novikova Jessica Zosa Forde Jordan Clive Jungo Kasai Ken Kawamura Liam Hazan Marine Carpuat Miruna Clinciu Najoung Kim Newton Cheng Oleg Serikov Omer Antverg Oskar van der Wal Rui Zhang Ruochen Zhang Sebastian Gehrmann Shachar Mirkin Shani Pais Tatiana Shavrina Thomas Scialom Tian Yun Tomasz Limisiewicz Verena Rieser Vitaly Protasov Vladislav Mikhailov Yada Pruksachatkun Yonatan Belinkov Zachary Bamberger Zden\u011bk Kasner Alice Rueda Amanda Pestana Amir Feizpour Ammar Khan Amy Faranak Ana Santos Anthony Hevia Antigona Unldreaj Arash Aghagol Arezoo Abdollahi Aycha Tammour Azadeh HajiHosseini Bahareh Behroozi Benjamin Ajibade Bharat Saxena Carlos Mu\u00f1oz Ferrandis Daniel McDuff Danish Contractor David Lansky Davis David Douwe Kiela Duong A. Nguyen Edward Tan Emi Baylor Ezinwanne Ozoani Fatima Mirza Frankline Ononiwu Habib Rezanejad Hessie Jones Indrani Bhattacharya Irene Solaiman Irina Sedenko Isar Nejadgholi Jesse Passmore Josh Seltzer Julio Bonis Sanz Livia Dutra Mairon Samagaio Maraim Elbadri Margot Mieskes Marissa Gerchick Martha Akinlolu Michael McKenna Mike Qiu Muhammed Ghauri Mykola Burynok Nafis Abrar Nazneen Rajani Nour Elkott Nour Fahmy Olanrewaju Samuel Ran An Rasmus Kromann Ryan Hao Samira Alizadeh Sarmad Shubber Silas Wang Sourav Roy Sylvain Viguier Thanh Le Tobi Oyebade Trieu Le Yoyo Yang Zach Nguyen Abhinav Ramesh Kashyap Alfredo Palasciano Alison Callahan Anima Shukla Antonio Miranda-Escalada Ayush Singh Benjamin Beilharz Bo Wang Caio Brito Chenxi Zhou Chirag Jain Chuxin Xu Cl\u00e9mentine Fourrier Daniel Le\u00f3n Peri\u00f1\u00e1n Daniel Molano Dian Yu Enrique Manjavacas Fabio Barth Florian Fuhrimann Gabriel Altay Giyaseddin Bayrak Gully Burns Helena U. Vrabec Imane Bello Ishani Dash Jihyun Kang John Giorgi Jonas Golde Jose David Posada Karthik Rangasai Sivaraman Lokesh Bulchandani Lu Liu Luisa Shinzato Madeleine Hahn de Bykhovetz Maiko Takeuchi Marc P\u00e0mies Maria A Castillo Marianna Nezhurina Mario S\u00e4nger Matthias Samwald Michael Cullan Michael Weinberg Michiel De Wolf Mina Mihaljcic Minna Liu Moritz Freidank Myungsun Kang Natasha Seelam Nathan Dahlberg Nicholas Michio Broad Nikolaus Muellner Pascale Fung Patrick Haller Ramya Chandrasekhar Renata Eisenberg Robert Martin Rodrigo Canalli Rosaline Su Ruisi Su Samuel Cahyawijaya Samuele Garda Shlok S Deshmukh Shubhanshu Mishra Sid Kiblawi Simon Ott Sinee Sang-aroonsiri Srishti Kumar Stefan Schweter Sushil Bharati Tanmay Laud Th\u00e9o Gigant Tomoya Kainuma Wojciech Kusa Yanis Labrak Yash Shailesh Bajaj Yash Venkatraman Yifan Xu Yingxin Xu Yu Xu Zhe Tan Zhongli Xie Zifan Ye Mathilde Bras Younes Belkada and Thomas Wolf. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022."},{"key":"e_1_3_4_45_2","unstructured":"Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen Shuohui Chen Christopher Dewan Mona Diab Xian Li Xi Victoria Lin Todor Mihaylov Myle Ott Sam Shleifer Kurt Shuster Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang and Luke Zettlemoyer. 2022. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 2022."},{"key":"e_1_3_4_46_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023."},{"key":"e_1_3_4_47_2","unstructured":"Tim Dettmers Artidoro Pagnoni Ari Holtzman and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. arXiv preprintarXiv:2305.14314 2023."},{"key":"e_1_3_4_48_2","doi-asserted-by":"crossref","unstructured":"Brian Lester Rami Al-Rfou and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 2021.","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_3_4_49_2","unstructured":"Yi-Lin Sung Jaemin Cho and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In Advances in Neural Information Processing Systems. 12991\u201313005."},{"key":"e_1_3_4_50_2","unstructured":"Zirui Liu Guanchu Wang Shaochen Zhong Zhaozhuo Xu Daochen Zha Ruixiang Tang Zhimeng Jiang Kaixiong Zhou Vipin Chaudhary Shuai Xu and Xia Hu. 2023. Winner-take-all column row sampling for memory efficient adaptation of language model. arXiv preprintarXiv:2305.15265 2023."},{"key":"e_1_3_4_51_2","doi-asserted-by":"crossref","unstructured":"Elad Ben Zaken Shauli Ravfogel and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 2021.","DOI":"10.18653\/v1\/2022.acl-short.1"},{"key":"e_1_3_4_52_2","unstructured":"Rabeeh Karimi Mahabadi James Henderson and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_4_53_2","first-page":"2790","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790\u20132799."},{"key":"e_1_3_4_54_2","first-page":"1273","volume-title":"Proceedings of the Artificial Intelligence and Statistics","author":"McMahan Brendan","year":"2017","unstructured":"Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the Artificial Intelligence and Statistics. PMLR, 1273\u20131282."},{"key":"e_1_3_4_55_2","first-page":"496","volume-title":"Proceedings of the Uncertainty in Artificial Intelligence","author":"Das Rudrajit","year":"2022","unstructured":"Rudrajit Das, Anish Acharya, Abolfazl Hashemi, Sujay Sanghavi, Inderjit S. Dhillon, and Ufuk Topcu. 2022. Faster non-convex federated learning via global and local momentum. In Proceedings of the Uncertainty in Artificial Intelligence. PMLR, 496\u2013506."},{"key":"e_1_3_4_56_2","unstructured":"Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar and Virginia Smith. 2020. Federated optimization in heterogeneous networks. In Machine Learning and Systems. 429\u2013450."},{"key":"e_1_3_4_57_2","first-page":"10334","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Woodworth Blake","year":"2020","unstructured":"Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. 2020. Is local SGD better than minibatch SGD?. In Proceedings of the International Conference on Machine Learning. PMLR, 10334\u201310343."},{"key":"e_1_3_4_58_2","first-page":"5132","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Karimireddy Sai Praneeth","year":"2020","unstructured":"Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the International Conference on Machine Learning. PMLR, 5132\u20135143."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2020.2988575"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978318"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1559845.1559850"},{"key":"e_1_3_4_62_2","doi-asserted-by":"crossref","unstructured":"Armen Aghajanyan Luke Zettlemoyer and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 2020.","DOI":"10.18653\/v1\/2021.acl-long.568"},{"key":"e_1_3_4_63_2","unstructured":"Together Computer. 2023. RedPajama: An Open Source Recipe to Reproduce LLaMA Training Dataset. (2023). https:\/\/github.com\/togethercomputer\/RedPajama-Data"},{"key":"e_1_3_4_64_2","unstructured":"Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai Ming Ding Zhuoyi Yang Yifan Xu Wendi Zheng Xiao Xia Weng Lam Tam Zixuan Ma Yufei Xue Jidong Zhai Wenguang Chen Peng Zhang Yuxiao Dong and Jie Tang. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 2022."},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.3041185"},{"key":"e_1_3_4_66_2","first-page":"954","volume-title":"Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems","author":"Luping Wang","year":"2019","unstructured":"Wang Luping, Wang Wei, and Li Bo. 2019. CMFL: Mitigating communication overhead for federated learning. In Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems. IEEE, 954\u2013964."}],"container-title":["ACM Transactions on Management Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3682068","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3682068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:10:03Z","timestamp":1750295403000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3682068"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,13]]},"references-count":65,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3682068"],"URL":"https:\/\/doi.org\/10.1145\/3682068","relation":{},"ISSN":["2158-656X","2158-6578"],"issn-type":[{"value":"2158-656X","type":"print"},{"value":"2158-6578","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,13]]},"assertion":[{"value":"2023-12-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}