{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T02:41:22Z","timestamp":1784342482721,"version":"3.55.0"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Large Code Models (LCMs) have demonstrated remarkable effectiveness across various code intelligence tasks. Supervised fine-tuning is essential to optimize their performance for specific downstream tasks. Compared with the traditional full-parameter fine-tuning (FFT) method, Parameter-Efficient Fine-Tuning (PEFT) methods can train LCMs with substantially reduced resource consumption and have gained widespread attention among researchers and practitioners. While existing studies have explored PEFT methods for code intelligence tasks, they have predominantly focused on a limited subset of scenarios, such as code generation with publicly available datasets, leading to constrained generalizability of the findings. To mitigate the limitation, we conduct a comprehensive study on exploring the effectiveness of the PEFT methods, which involves five code intelligence tasks containing both public and private data. Our extensive experiments reveal a considerable performance gap between PEFT methods and FFT, which is contrary to the findings of existing studies. We also find that this disparity is particularly pronounced in tasks involving private data.<\/jats:p>\n          <jats:p>To improve the tuning performance for LCMs while reducing resource utilization during training, we propose a Layer-Wise Optimization (LWO) strategy in the paper. LWO incrementally updates the parameters of each layer in the whole model architecture, without introducing any additional component and inference overhead. Experiments across five LCMs and five code intelligence tasks demonstrate that LWO trains LCMs more effectively and efficiently compared to previous PEFT methods, with significant improvements in tasks using private data. For instance, in the line-level code completion task using our private code repositories, LWO outperforms the state-of-the-art LoRA method by 22% and 12% in terms of accuracy and BLEU scores, respectively. Furthermore, LWO can enable more efficient LCM tuning, reducing the training time by an average of 42.7% compared to LoRA.<\/jats:p>","DOI":"10.1145\/3729341","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:15:34Z","timestamp":1750346134000},"page":"1567-1590","source":"Crossref","is-referenced-by-count":5,"title":["Beyond PEFT: Layer-Wise Optimization for More Effective and Efficient Large Code Model Tuning"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3935-7328","authenticated-orcid":false,"given":"Chaozheng","family":"Wang","sequence":"first","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5509-4724","authenticated-orcid":false,"given":"Jia","family":"Feng","sequence":"additional","affiliation":[{"name":"University of Electronic Science and Technology of China, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8102-480X","authenticated-orcid":false,"given":"Shuzheng","family":"Gao","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8513-6836","authenticated-orcid":false,"given":"Cuiyun","family":"Gao","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9897-4086","authenticated-orcid":false,"given":"Zongjie","family":"Li","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6970-0857","authenticated-orcid":false,"given":"Ting","family":"Peng","sequence":"additional","affiliation":[{"name":"Tencent, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-0655-9398","authenticated-orcid":false,"given":"Hailiang","family":"Huang","sequence":"additional","affiliation":[{"name":"Tencent, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-7060-4109","authenticated-orcid":false,"given":"Yuetang","family":"Deng","sequence":"additional","affiliation":[{"name":"Tencent, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3666-5798","authenticated-orcid":false,"given":"Michael","family":"Lyu","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n. d.]. Copilot. https:\/\/github.com\/features\/copilot"},{"key":"e_1_2_1_2_1","unstructured":"[n. d.]. tabnine. https:\/\/www.tabnine.com\/"},{"key":"e_1_2_1_3_1","unstructured":"Toufique Ahmed Christian Bird Premkumar Devanbu and Saikat Chakraborty. 2024. Studying LLM Performance on Closed-and Open-source Data. arXiv preprint arXiv:2402.15100."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, ICSE 2024","author":"Ahmed Toufique","year":"2024","unstructured":"Toufique Ahmed, Kunal Suresh Pai, Premkumar T. Devanbu, and Earl T. Barr. 2024. Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 220:1\u2013220:13."},{"key":"e_1_2_1_5_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry and Quoc Le. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Manish Bhattarai Javier E Santos Shawn Jones Ayan Biswas Boian Alexandrov and Daniel O\u2019Malley. 2024. Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation. arXiv preprint arXiv:2407.19619.","DOI":"10.1109\/HPEC62836.2024.10938485"},{"key":"e_1_2_1_7_1","volume-title":"Brown and et al","author":"Tom","year":"2020","unstructured":"Tom B. Brown and et al.. 2020. Language Models are Few-Shot Learners. ArXiv, abs\/2005.14165 (2020), https:\/\/api.semanticscholar.org\/CorpusID:218971783"},{"key":"e_1_2_1_8_1","unstructured":"Liguo Chen Qi Guo Hongrui Jia Zhengran Zeng Xin Wang Yijiang Xu Jian Wu Yidong Wang Qing Gao and Jindong Wang. 2024. A Survey on Evaluating Large Language Models in Code Generation Tasks. arXiv preprint arXiv:2408.16498."},{"key":"e_1_2_1_9_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374."},{"key":"e_1_2_1_10_1","volume-title":"Vullibgen: Identifying vulnerable third-party libraries via generative pre-trained model. arXiv preprint arXiv:2308.04662.","author":"Chen Tianyu","year":"2023","unstructured":"Tianyu Chen, Lin Li, Liuchuan Zhu, Zongyang Li, Guangtai Liang, Ding Li, Qianxiang Wang, and Tao Xie. 2023. Vullibgen: Identifying vulnerable third-party libraries via generative pre-trained model. arXiv preprint arXiv:2308.04662."},{"key":"e_1_2_1_11_1","unstructured":"Tianqi Chen Bing Xu Chiyuan Zhang and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174."},{"key":"e_1_2_1_12_1","first-page":"1","article-title":"Scaling instruction-finetuned language models","volume":"25","author":"Chung Hyung Won","year":"2024","unstructured":"Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, and Siddhartha Brahma. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25, 70 (2024), 1\u201353.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_13_1","first-page":"16344","article-title":"Flashattention: Fast and memory-efficient exact attention with io-awareness","volume":"35","author":"Dao Tri","year":"2022","unstructured":"Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R\u00e9. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35 (2022), 16344\u201316359.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1038\/s42256-023-00626-4","article-title":"Parameter-efficient fine-tuning of large-scale pre-trained language models","volume":"5","author":"Ding Ning","year":"2023","unstructured":"Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, and Weize Chen. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5, 3 (2023), 220\u2013235.","journal-title":"Nature Machine Intelligence"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Sujan Dutta Sayantan Mahinder Raviteja Anantha and Bortik Bandyopadhyay. 2024. Applying RLAIF for Code Generation with API-usage in Lightweight LLMs. arXiv preprint arXiv:2406.20060.","DOI":"10.18653\/v1\/2024.nlrse-1.4"},{"key":"e_1_2_1_16_1","unstructured":"Angela Fan Beliz Gokkaya Mark Harman Mitya Lyubarskiy Shubho Sengupta Shin Yoo and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533."},{"key":"e_1_2_1_17_1","volume-title":"Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155.","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155."},{"key":"e_1_2_1_18_1","volume-title":"Dipti Misra Sharma, and Vasudeva Varma","author":"Gahoi Akshat","year":"2022","unstructured":"Akshat Gahoi, Jayant Duneja, Anshul Padhi, Shivam Mangale, Saransh Rajput, Tanvi Kamble, Dipti Misra Sharma, and Vasudeva Varma. 2022. Gui at MixMT 2022: English-Hinglish: An MT approach for translation of code mixed data. arXiv preprint arXiv:2210.12215."},{"key":"e_1_2_1_19_1","volume-title":"2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). 254\u2013266","author":"Gao Shuzheng","year":"2024","unstructured":"Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2024. Search-based llms for code optimization. In 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). 254\u2013266."},{"key":"e_1_2_1_20_1","article-title":"Code Structure-Guided Transformer for Source Code Summarization","volume":"32","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nie, Xin Xia, and Michael R. Lyu. 2023. Code Structure-Guided Transformer for Source Code Summarization. ACM Trans. Softw. Eng. Methodol., 32, 1 (2023), 23:1\u201323:32.","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, ICSE 2024","author":"Gao Shuzheng","year":"2024","unstructured":"Shuzheng Gao, Wenxin Mao, Cuiyun Gao, Li Li, Xing Hu, Xin Xia, and Michael R. Lyu. 2024. Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 80:1\u201380:13."},{"key":"e_1_2_1_22_1","volume-title":"2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 761\u2013773","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. 2023. What makes good in-context demonstrations for code intelligence tasks with llms? In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 761\u2013773."},{"key":"e_1_2_1_23_1","volume-title":"Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models. In 45th IEEE\/ACM International Conference on Software Engineering, ICSE 2023","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Hongyu Zhang, Cuiyun Gao, and Chaozheng Wang. 2023. Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models. In 45th IEEE\/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 30\u201342."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 5484\u20135495","author":"Geva Mor","year":"2021","unstructured":"Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 5484\u20135495."},{"key":"e_1_2_1_25_1","volume-title":"Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha.","author":"Ghosh Sreyan","year":"2024","unstructured":"Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. 2024. A Closer Look at the Limitations of Instruction Tuning. arXiv preprint arXiv:2402.05119."},{"key":"e_1_2_1_26_1","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y Wu and YK Li. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming\u2013The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track). 133\u2013144","author":"Hong Kung Yin","year":"2024","unstructured":"Kung Yin Hong, Lifeng Han, Riza Theresa Batista-Navarro, and Goran Nenadic. 2024. CantonMT: Cantonese-English neural machine translation looking into evaluations. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track). 133\u2013144."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Marko Hostnik and Marko Robnik-\u0160ikonja. 2024. Retrieval-augmented code completion for local projects using large language models. arXiv preprint arXiv:2408.05026.","DOI":"10.1016\/j.eswa.2025.128596"},{"key":"e_1_2_1_29_1","volume-title":"International conference on machine learning. 2790\u20132799","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. 2790\u20132799."},{"key":"e_1_2_1_30_1","volume-title":"Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.","author":"Hu Edward J","year":"2021","unstructured":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685."},{"key":"e_1_2_1_31_1","volume-title":"Decoupled Weight Decay Regularization. International Conference on Learning Representations, ICLR.","author":"Ilya Loshchilov","year":"2018","unstructured":"Loshchilov Ilya and Hutter Frank. 2018. Decoupled Weight Decay Regularization. International Conference on Learning Representations, ICLR."},{"key":"e_1_2_1_32_1","unstructured":"Zhenlan Ji Pingchuan Ma Zongjie Li and Shuai Wang. 2023. Benchmarking and explaining large language model-based code generation: A causality-centric approach. arXiv preprint arXiv:2310.06680."},{"key":"e_1_2_1_33_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 29th Symposium on Operating Systems Principles. 611\u2013626","author":"Kwon Woosuk","year":"2023","unstructured":"Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611\u2013626."},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Brian Lester Rami Al-Rfou and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_2_1_36_1","article-title":"StarCoder: may the source be with you!","author":"Li Raymond","year":"2023","unstructured":"Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, LI Jia, Jenny Chim, and Qian Liu. 2023. StarCoder: may the source be with you!. Transactions on Machine Learning Research.","journal-title":"Transactions on Machine Learning Research."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING","author":"Li Taiji","year":"2024","unstructured":"Taiji Li, Zhi Li, and Yin Zhang. 2024. Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 8804\u20138817."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.112002"},{"key":"e_1_2_1_39_1","volume-title":"Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.","author":"Li Xiang Lisa","year":"2021","unstructured":"Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190."},{"key":"e_1_2_1_40_1","volume-title":"Competition-level code generation with alphacode. Science, 378, 6624","author":"Li Yujia","year":"2022","unstructured":"Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\u00e9mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, and Agustin Dal Lago. 2022. Competition-level code generation with alphacode. Science, 378, 6624 (2022), 1092\u20131097."},{"key":"e_1_2_1_41_1","volume-title":"CCTEST: Testing and Repairing Code Completion Systems. In 45th IEEE\/ACM International Conference on Software Engineering, ICSE 2023","author":"Li Zongjie","year":"2023","unstructured":"Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023. CCTEST: Testing and Repairing Code Completion Systems. In 45th IEEE\/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1238\u20131250."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE \u201924)","author":"Li Zongjie","year":"2024","unstructured":"Zongjie Li, Chaozheng Wang, Pingchuan Ma, Chaowei Liu, Shuai Wang, Daoyuan Wu, Cuiyun Gao, and Yang Liu. 2024. On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE \u201924). Association for Computing Machinery, New York, NY, USA. Article 74, 13 pages."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.).","author":"Li Zongjie","year":"2024","unstructured":"Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024. Split and Merge: Aligning Position Biases in LLM-based Evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.)."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023","author":"Li Zongjie","year":"2023","unstructured":"Zongjie Li, Chaozheng Wang, Shuai Wang, and Gao Cuiyun. 2023. Protecting Intellectual Property of Large Language Model-Based Code Generation APIs via Watermarks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023."},{"key":"e_1_2_1_45_1","unstructured":"Zongjie Li Daoyuan Wu Shuai Wang and Zhendong Su. 2024. API-guided Dataset Synthesis to Finetune Large Code Models. arXiv preprint arXiv:2408.08343."},{"key":"e_1_2_1_46_1","unstructured":"Bingchang Liu Chaoyu Chen Cong Liao Zi Gong Huan Wang Zhichao Lei Ming Liang Dajun Chen Min Shen Hailian Zhou Hang Yu and Jianguo Li. 2023. MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning. arXiv preprint arXiv arxiv:2311.02303."},{"key":"e_1_2_1_47_1","first-page":"1950","article-title":"Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning","volume":"35","author":"Liu Haokun","year":"2022","unstructured":"Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35 (2022), 1950\u20131965.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_48_1","volume-title":"Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=1qvx610Cu7","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=1qvx610Cu7"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 18266\u201318287","author":"Liu Yongkang","year":"2024","unstructured":"Yongkang Liu, Yiqun Zhang, Qian Li, Tong Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Sch\u00fctze. 2024. HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 18266\u201318287."},{"key":"e_1_2_1_50_1","volume-title":"SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations.","author":"Loshchilov Ilya","year":"2016","unstructured":"Ilya Loshchilov and Frank Hutter. 2016. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations."},{"key":"e_1_2_1_51_1","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei Tianyang Liu Max Tian Denis Kocetkov Arthur Zucker Younes Belkada Zijian Wang Qian Liu Dmitry Abulkhanov Indraneil Paul Zhuang Li Wen-Ding Li Megan Risdal Jia Li Jian Zhu Terry Yue Zhuo Evgenii Zheltonozhskii Nii Osae Osae Dade Wenhao Yu Lucas Krau\u00df Naman Jain Yixuan Su Xuanli He Manan Dey Edoardo Abati Yekun Chai Niklas Muennighoff Xiangru Tang Muhtasham Oblokulov Christopher Akiki Marc Marone Chenghao Mou Mayank Mishra Alex Gu Binyuan Hui Tri Dao Armel Zebaze Olivier Dehaene Nicolas Patry Canwen Xu Julian J. McAuley Han Hu Torsten Scholak S\u00e9bastien Paquet Jennifer Robinson Carolyn Jane Anderson Nicolas Chapados and et al.. 2024. StarCoder 2 and The Stack v2: The Next Generation. CoRR abs\/2402.19173 (2024)."},{"key":"e_1_2_1_52_1","volume-title":"Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664.","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, and Duyu Tang. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664."},{"key":"e_1_2_1_53_1","first-page":"24926","article-title":"BAdam: A memory efficient full parameter optimization method for large language models","volume":"37","author":"Luo Qijun","year":"2024","unstructured":"Qijun Luo, Hengxu Yu, and Xiao Li. 2024. BAdam: A memory efficient full parameter optimization method for large language models. Advances in Neural Information Processing Systems, 37 (2024), 24926\u201324958.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_54_1","volume-title":"WizardCoder: Empowering Code Large Language Models with Evol-Instruct. In The Twelfth International Conference on Learning Representations.","author":"Luo Ziyang","year":"2023","unstructured":"Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_2_1_55_1","volume-title":"PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https:\/\/github.com\/huggingface\/peft","author":"Mangrulkar Sourab","year":"2022","unstructured":"Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https:\/\/github.com\/huggingface\/peft"},{"key":"e_1_2_1_56_1","unstructured":"Meta AI. 2025. Llama 3.2 Model Cards and Prompt Formats. https:\/\/www.llama.com\/docs\/model-cards-and-prompt-formats\/llama3_2\/"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, ICSE 2024","author":"Peng Yun","year":"2024","unstructured":"Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, and Michael R. Lyu. 2024. Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 4:1\u20134:13."},{"key":"e_1_2_1_58_1","volume-title":"Generative Type Inference for Python. In 38th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2023","author":"Peng Yun","year":"2023","unstructured":"Yun Peng, Chaozheng Wang, Wenxuan Wang, Cuiyun Gao, and Michael R. Lyu. 2023. Generative Type Inference for Python. In 38th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 988\u2013999."},{"key":"e_1_2_1_59_1","volume-title":"Language models are unsupervised multitask learners. OpenAI blog, 1, 8","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1, 8 (2019), 9."},{"key":"e_1_2_1_60_1","volume-title":"SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. 1\u201316","author":"Rajbhandari Samyam","year":"2020","unstructured":"Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. 1\u201316."},{"key":"e_1_2_1_61_1","volume-title":"2021 USENIX Annual Technical Conference (USENIX ATC 21)","author":"Ren Jie","year":"2021","unstructured":"Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. $Zero-offload$: Democratizing $billion-scale$ model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 551\u2013564."},{"key":"e_1_2_1_62_1","volume-title":"Yossi Adi, Jingyu Liu, Tal Remez, and J\u00e9r\u00e9my Rapin.","author":"Rozi\u00e8re Baptiste","year":"2023","unstructured":"Baptiste Rozi\u00e8re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, and J\u00e9r\u00e9my Rapin. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950."},{"key":"e_1_2_1_63_1","volume-title":"Yossi Adi, Jingyu Liu, Tal Remez, and J\u00e9r\u00e9my Rapin.","author":"Roziere Baptiste","year":"2023","unstructured":"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, and J\u00e9r\u00e9my Rapin. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950."},{"key":"e_1_2_1_64_1","unstructured":"Bo Shen Jiaxin Zhang Taihong Chen Daoguang Zan Bing Geng An Fu Muhan Zeng Ailun Yu Jichuan Ji and Jingyang Zhao. 2023. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 39\u201351","author":"Shi Ensheng","year":"2023","unstructured":"Ensheng Shi, Yanlin Wang, Hongyu Zhang, Lun Du, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 39\u201351."},{"key":"e_1_2_1_66_1","volume-title":"IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER","author":"Shuo LIU","year":"2023","unstructured":"LIU Shuo, Jacky Keung, YANG Zhen, Fang Liu, ZHOU Qilin, and LIAO Yihan. 2023. Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study. In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2024)."},{"key":"e_1_2_1_67_1","unstructured":"Reece Shuttleworth Jacob Andreas Antonio Torralba and Pratyusha Sharma. 2024. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228."},{"key":"e_1_2_1_68_1","volume-title":"Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4, 2","author":"Tieleman Tijmen","year":"2012","unstructured":"Tijmen Tieleman. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4, 2 (2012), 26."},{"key":"e_1_2_1_69_1","unstructured":"Hugo Touvron and et al.. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv abs\/2307.09288 (2023) https:\/\/api.semanticscholar.org\/CorpusID:259950998"},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2024","author":"Wang Chaozheng","year":"2024","unstructured":"Chaozheng Wang, Shuzheng Gao, Cuiyun Gao, Wenxuan Wang, Chun Yong Chong, Shan Gao, and Michael R. Lyu. 2024. A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024. ACM, 281\u2013293."},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1294\u20131306","author":"Wang Chaozheng","year":"2023","unstructured":"Chaozheng Wang, Junhao Hu, Cuiyun Gao, Yu Jin, Tao Xie, Hailiang Huang, Zhenyu Lei, and Yuetang Deng. 2023. How Practitioners Expect Code Completion? In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1294\u20131306."},{"key":"e_1_2_1_72_1","unstructured":"Chaozheng Wang Zongjie Li Cuiyun Gao Wenxuan Wang Ting Peng Hailiang Huang Yuetang Deng Shuai Wang and Michael R Lyu. 2024. Exploring Multi-Lingual Bias of Large Code Models in Code Generation. arXiv preprint arXiv:2404.19368."},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 382\u2013394","author":"Wang Chaozheng","year":"2022","unstructured":"Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2022. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 382\u2013394."},{"key":"e_1_2_1_74_1","doi-asserted-by":"crossref","unstructured":"Chong Wang Jian Zhang Yebo Feng Tianlin Li Weisong Sun Yang Liu and Xin Peng. 2024. Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation. arXiv preprint arXiv:2401.06391.","DOI":"10.1145\/3714462"},{"key":"e_1_2_1_75_1","volume-title":"Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120.","author":"Wei Yuxiang","year":"2023","unstructured":"Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120."},{"key":"e_1_2_1_76_1","unstructured":"Martin Weyssow Xin Zhou Kisub Kim David Lo and Houari Sahraoui. 2023. Exploring parameter-efficient fine-tuning techniques for code generation with large language models. arXiv preprint arXiv:2308.10462."},{"key":"e_1_2_1_77_1","unstructured":"Wai Kin Wong Huaijin Wang Zongjie Li Zhibo Liu Shuai Wang Qiyi Tang Sen Nie and Shi Wu. 2023. Refining decompiled c code with large language models. arXiv preprint arXiv:2310.06530."},{"key":"e_1_2_1_78_1","volume-title":"Xiaohui Tao, and Fu Lee Wang.","author":"Xu Lingling","year":"2023","unstructured":"Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148."},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 5140\u20135153","author":"Yu Zhaojian","year":"2024","unstructured":"Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2024. WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 5140\u20135153."},{"key":"e_1_2_1_80_1","unstructured":"Kunpeng Zhang Zongjie Li Daoyuan Wu Shuai Wang and Xin Xia. 2025. Low-Cost and Comprehensive Non-textual Input Fuzzing with LLM-Synthesized Input Generators. arXiv preprint arXiv:2501.19282."},{"key":"e_1_2_1_81_1","volume-title":"Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199.","author":"Zhang Renrui","year":"2023","unstructured":"Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199."},{"key":"e_1_2_1_82_1","volume-title":"2018 IEEE\/ACM 26th international symposium on quality of service (IWQoS). 1\u20132.","author":"Zhang Zijun","year":"2018","unstructured":"Zijun Zhang. 2018. Improved adam optimizer for deep neural networks. In 2018 IEEE\/ACM 26th international symposium on quality of service (IWQoS). 1\u20132."}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3729341","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:30:42Z","timestamp":1750347042000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729341"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":82,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3729341"],"URL":"https:\/\/doi.org\/10.1145\/3729341","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}