{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T19:52:36Z","timestamp":1795204356163,"version":"4.10.3"},"reference-count":333,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,2,28]]},"abstract":"<jats:p>\n                    Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g.,\n                    <jats:italic toggle=\"yes\">GitHub Copilot<\/jats:italic>\n                    . Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of Natural Language Processing (NLP) or Software Engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and provide a quantitative and qualitative comparative analysis of experimental results of code LLMs, sourced from their original papers to ensure a fair comparison on the HumanEval, MBPP, and BigCodeBench benchmarks, across various levels of difficulty and types of programming tasks, to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page (\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/juyongjiang\/CodeLLMSurvey\">https:\/\/github.com\/juyongjiang\/CodeLLMSurvey<\/jats:ext-link>\n                    ) to continuously document and disseminate the most recent advances in the field.\n                  <\/jats:p>","DOI":"10.1145\/3747588","type":"journal-article","created":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T09:25:42Z","timestamp":1751966742000},"page":"1-72","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":248,"title":["A Survey on Large Language Models for Code Generation"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0835-9686","authenticated-orcid":false,"given":"Juyong","family":"Jiang","sequence":"first","affiliation":[{"name":"The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9447-9496","authenticated-orcid":false,"given":"Fan","family":"Wang","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5904-3641","authenticated-orcid":false,"given":"Jiasi","family":"Shen","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2123-2925","authenticated-orcid":false,"given":"Sungju","family":"Kim","sequence":"additional","affiliation":[{"name":"NAVER Cloud, Seoul, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4478-9720","authenticated-orcid":false,"given":"Sunghun","family":"Kim","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,1,21]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"GitHub. 2023. AgentGPT: Assemble Configure and Deploy Autonomous AI Agents in Your Browser. Retrieved from https:\/\/github.com\/reworkd\/AgentGPT"},{"key":"e_1_3_2_3_2","unstructured":"GitHub. 2023. AutoGPT Is the Vision of Accessible AI for Everyone to Use and to Build On. Retrieved from https:\/\/github.com\/Significant-Gravitas\/AutoGPT"},{"key":"e_1_3_2_4_2","unstructured":"GitHub. 2023. BabyAGI. Retrieved from https:\/\/github.com\/yoheinakajima\/babyagi"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.02271"},{"key":"e_1_3_2_6_2","unstructured":"Marah Abdin Sam Ade Jacobs Ammar Ahmad Awan Jyoti Aneja Ahmed Awadallah Hany Awadalla Nguyen Bach Amit Bahree Arash Bakhtiari Harkirat Behl et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219. Retrieved from https:\/\/arxiv.org\/abs\/2404.14219"},{"key":"e_1_3_2_7_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.449"},{"key":"e_1_3_2_9_2","unstructured":"Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv:2103.06333. Retrieved from https:\/\/arxiv.org\/abs\/2103.06333"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639133"},{"key":"e_1_3_2_11_2","unstructured":"Loubna Ben Allal Raymond Li Denis Kocetkov Chenghao Mou Christopher Akiki Carlos Munoz Ferrandis Niklas Muennighoff Mayank Mishra Alex Gu Manan Dey et al. 2023. SantaCoder: Don\u2019t reach for the stars! arXiv:2301.03988. Retrieved from https:\/\/arxiv.org\/abs\/2301.03988"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635901"},{"key":"e_1_3_2_13_2","unstructured":"Google DeepMind AlphaCode Team. 2023. AlphaCode 2 Technical Report. Retrieved from https:\/\/storage.googleapis.com\/deepmind-media\/AlphaCode2\/AlphaCode2_Tech_Report.pdf"},{"key":"e_1_3_2_14_2","unstructured":"Amazon. 2022. What Is CodeWhisperer? https:\/\/docs.aws.amazon.com\/codewhisperer\/latest\/userguide\/what-is-cwspr.html"},{"key":"e_1_3_2_15_2","first-page":"2357","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Amini Aida","year":"2019","unstructured":"Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2357\u20132367."},{"key":"e_1_3_2_16_2","unstructured":"Anthropic. 2024. The Claude 3 Model Family: Opus Sonnet Haiku. Retrieved from https:\/\/www-cdn.anthropic.com\/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627\/Model_Card_Claude_3.pdf"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/8840389"},{"key":"e_1_3_2_18_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Athiwaratkun Ben","year":"2022","unstructured":"Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_19_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_20_2","unstructured":"Jimmy Lei Ba Jamie Ryan Kiros and Geoffrey E. Hinton. 2016. Layer normalization. arXiv:1607.06450. Retrieved from https:\/\/arxiv.org\/abs\/1607.06450"},{"key":"e_1_3_2_21_2","unstructured":"Hannah McLean Babe Sydney Nguyen Yangtian Zi Arjun Guha Molly Q. Feldman and Carolyn Jane Anderson. 2023. StudentEval: A benchmark of student-written prompts for large language models of code. arXiv:2306.04556. Retrieved from https:\/\/arxiv.org\/abs\/2306.04556"},{"key":"e_1_3_2_22_2","unstructured":"Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang Xiaodong Deng Yang Fan Wenbin Ge Yu Han Fei Huang et al. 2023. Qwen technical report. arXiv:2309.16609. Retrieved from https:\/\/arxiv.org\/abs\/2309.16609"},{"key":"e_1_3_2_23_2","unstructured":"Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell Jackson Kernion Andy Jones Anna Chen Anna Goldie Azalia Mirhoseini Cameron McKinnon et al. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. Retrieved from https:\/\/arxiv.org\/abs\/2212.08073"},{"key":"e_1_3_2_24_2","unstructured":"Ramakrishna Bairi Atharv Sonwane Aditya Kanade Arun Iyer Suresh Parthasarathy Sriram Rajamani B. Ashok and Shashank Shet. 2023. CodePlan: Repository-level coding using LLMs and planning. arXiv:2309.12499. Retrieved from https:\/\/arxiv.org\/abs\/2309.12499"},{"key":"e_1_3_2_25_2","first-page":"65","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, 65\u201372."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12094619"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586030"},{"key":"e_1_3_2_28_2","unstructured":"Xiao Bi Deli Chen Guanting Chen Shanhuang Chen Damai Dai Chengqi Deng Honghui Ding Kai Dong Qiushi Du Zhe Fu et al. 2024. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954. Retrieved from https:\/\/arxiv.org\/abs\/2401.02954"},{"key":"e_1_3_2_29_2","unstructured":"Zhangqian Bi Yao Wan Zheng Wang Hongyu Zhang Batu Guan Fangxin Lu Zili Zhang Yulei Sui Xuanhua Shi and Hai Jin. 2024. Iterative refinement of project-level code context for precise code generation with compiler feedback. arXiv:2403.16792. Retrieved from https:\/\/arxiv.org\/abs\/2403.16792"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582083"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","unstructured":"Sid Black Stella Biderman Eric Hallahan Quentin Anthony Leo Gao Laurence Golding Horace He Connor Leahy Kyle McDonell Jason Phang et al. 2022. GPT-NeoX-20B: An open-source autoregressive language model. arXiv:2204.06745. Retrieved from https:\/\/arxiv.org\/abs\/2204.06745","DOI":"10.18653\/v1\/2022.bigscience-1.9"},{"key":"e_1_3_2_32_2","unstructured":"Sid Black Leo Gao Phil Wang Connor Leahy and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. Zenodo. Retrieved March 21 2021 https:\/\/zenodo.org\/records\/5297715"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-021-00248-0"},{"key":"e_1_3_2_34_2","unstructured":"Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill et al. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258. Retrieved from https:\/\/arxiv.org\/abs\/2108.07258"},{"key":"e_1_3_2_35_2","unstructured":"Tom B. Brown. 2020. Language models are few-shot learners. arXiv:2005.14165. Retrieved from https:\/\/arxiv.org\/abs\/2005.14165"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.70"},{"key":"e_1_3_2_37_2","unstructured":"Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim and Jiayi Huang .2024. Shortcut-connected expert parallelism for accelerating mixture-of-experts. arXiv:2404.05019. Retrieved from https:\/\/arxiv.org\/abs\/2404.05019"},{"key":"e_1_3_2_38_2","unstructured":"Weilin Cai Juyong Jiang Fan Wang Jing Tang Sunghun Kim and Jiayi Huang. 2024. A survey on mixture of experts. arXiv:2407.06204. Retrieved from https:\/\/arxiv.org\/abs\/2407.06204"},{"key":"e_1_3_2_39_2","first-page":"2633","volume-title":"Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921)","author":"Carlini Nicholas","year":"2021","unstructured":"Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921), 2633\u20132650."},{"key":"e_1_3_2_40_2","unstructured":"Federico Cassano John Gouwar Francesca Lucchetti Claire Schlesinger Carolyn Jane Anderson Michael Greenberg Abhinav Jangda and Arjun Guha. 2023. Knowledge transfer from high-resource to low-resource programming languages for code LLMs. arXiv:2308.09895. Retrieved from https:\/\/arxiv.org\/abs\/2308.09895"},{"key":"e_1_3_2_41_2","unstructured":"Federico Cassano John Gouwar Daniel Nguyen Sydney Nguyen Luna Phipps-Costin Donald Pinckney Ming-Ho Yee Yangtian Zi Carolyn Jane Anderson Molly Q. Feldman et al. 2022. A scalable and extensible approach to benchmarking NL2Code for 18 programming languages. arXiv:2208.08227. Retrieved from https:\/\/arxiv.org\/abs\/2208.08227"},{"key":"e_1_3_2_42_2","unstructured":"Yekun Chai Shuohuan Wang Chao Pang Yu Sun Hao Tian and Hua Wu. 2022. ERNIE-Code: Beyond English-centric cross-lingual pretraining for programming languages. arXiv:2212.06742. Retrieved from https:\/\/arxiv.org\/abs\/2212.06742"},{"key":"e_1_3_2_43_2","unstructured":"Shubham Chandel Colin B. Clement Guillermo Serrato and Neel Sundaresan. 2022. Training and evaluating a Jupyter notebook data science assistant. arXiv:2201.12901. Retrieved from https:\/\/arxiv.org\/abs\/2201.12901"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_3_2_45_2","unstructured":"Sahil Chaudhary. 2023. Code Alpaca: An Instruction-Following LLaMA Model for Code Generation. Retrieved from https:\/\/github.com\/sahil280114\/codealpaca"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00198"},{"key":"e_1_3_2_47_2","unstructured":"Bei Chen Fengji Zhang Anh Nguyen Daoguang Zan Zeqi Lin Jian-Guang Lou and Weizhu Chen. 2022. CodeT: Code generation with generated tests. arXiv:2207.10397. Retrieved from https:\/\/arxiv.org\/abs\/2207.10397"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524610.3527917"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i16.29728"},{"key":"e_1_3_2_50_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","unstructured":"Stanley F. Chen Douglas Beeferman and Roni Rosenfeld. 2018. Evaluation metrics for language models. DOI: 10.1184\/R1\/6605324.v1","DOI":"10.1184\/R1\/6605324.v1"},{"key":"e_1_3_2_52_2","unstructured":"Wenhu Chen Xueguang Ma Xinyi Wang and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv:2211.12588. Retrieved from https:\/\/arxiv.org\/abs\/2211.12588"},{"key":"e_1_3_2_53_2","unstructured":"Xinyun Chen Maxwell Lin Nathanael Sch\u00e4rli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv:2304.05128. Retrieved from https:\/\/arxiv.org\/abs\/2304.05128"},{"key":"e_1_3_2_54_2","article-title":"Tree-to-tree neural networks for program translation","volume":"31","author":"Chen Xinyun","year":"2018","unstructured":"Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 31.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604930.3605705"},{"issue":"240","key":"e_1_3_2_56_2","first-page":"1","article-title":"PaLM: Scaling language modeling with pathways","volume":"24","author":"Chowdhery Aakanksha","year":"2023","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1\u2013113.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_57_2","unstructured":"Fenia Christopoulou Gerasimos Lampouras Milan Gritta Guchun Zhang Yinpeng Guo Zhongqi Li Qi Zhang Meng Xiao Bo Shen Lin Li et al. 2022. PanGu-Coder: Program synthesis with function-level language modeling. arXiv:2207.11280. Retrieved from https:\/\/arxiv.org\/abs\/2207.11280"},{"issue":"70","key":"e_1_3_2_58_2","first-page":"1","article-title":"Scaling instruction-finetuned language models","volume":"25","author":"Chung Hyung Won","year":"2024","unstructured":"Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25, 70 (2024), 1\u201353.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","unstructured":"Colin B. Clement Dawn Drain Jonathan Timcheck Alexey Svyatkovskiy and Neel Sundaresan. 2020. PyMT5: Multi-mode translation of natural language and Python code with transformers. arXiv:2010.03150. Retrieved from https:\/\/arxiv.org\/abs\/2010.03150","DOI":"10.18653\/v1\/2020.emnlp-main.728"},{"key":"e_1_3_2_60_2","unstructured":"Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training verifiers to solve math word problems. arXiv:2110.14168. Retrieved from https:\/\/arxiv.org\/abs\/2110.14168"},{"key":"e_1_3_2_61_2","unstructured":"CodeGemma Team Ale Jakse Hartman Andrea Hu Christopher A. Choquette-Choo Heri Zhao Jane Fine Jeffrey Hui Jingyue Shen Joe Kelley Joshua Howland et al. Gong. 2024. CodeGemma: Open Code Models Based on Gemma. Retrieved from https:\/\/goo.gle\/codegemma"},{"key":"e_1_3_2_62_2","unstructured":"Codeium. 2023. Free Ultrafast Copilot Alternative for Vim and Neovim. Retrieved from https:\/\/github.com\/Exafunction\/codeium.vim"},{"key":"e_1_3_2_63_2","unstructured":"Cognition. 2024. Introducing Devin the First AI Software Engineer. Retrieved from https:\/\/www.cognition.ai\/introducing-devin"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1953031"},{"key":"e_1_3_2_65_2","unstructured":"Cognitive Computations. 2023. oa_leet10k. Retrieved from https:\/\/huggingface.co\/datasets\/cognitivecomputations\/oa_leet10k"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.5555\/1792734.1792766"},{"key":"e_1_3_2_67_2","article-title":"QLoRA: Efficient finetuning of quantized LLMs","volume":"36","author":"Dettmers Tim","year":"2024","unstructured":"Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. QLoRA: Efficient finetuning of quantized LLMs. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_68_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https:\/\/arxiv.org\/abs\/1810.04805"},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","unstructured":"Ning Ding Yujia Qin Guang Yang Fuchao Wei Zonghan Yang Yusheng Su Shengding Hu Yulin Chen Chi-Min Chan Weize Chen et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv:2203.06904. Retrieved from https:\/\/arxiv.org\/abs\/2203.06904","DOI":"10.21203\/rs.3.rs-1553541\/v1"},{"key":"e_1_3_2_70_2","article-title":"CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion","volume":"36","author":"Ding Yangruibo","year":"2024","unstructured":"Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2024. CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_71_2","unstructured":"Qingxiu Dong Lei Li Damai Dai Ce Zheng Zhiyong Wu Baobao Chang Xu Sun Jingjing Xu and Zhifang Sui. 2022. A survey on in-context learning. arXiv:2301.00234. Retrieved from https:\/\/arxiv.org\/abs\/2301.00234"},{"key":"e_1_3_2_72_2","unstructured":"Shihan Dou Yan Liu Haoxiang Jia Limao Xiong Enyu Zhou Junjie Shan Caishuang Huang Wei Shen Xiaoran Fan Zhiheng Xi et al. 2024. StepCoder: Improve code generation with reinforcement learning from compiler feedback. arXiv:2402.01391. Retrieved from https:\/\/arxiv.org\/abs\/2402.01391"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639219"},{"key":"e_1_3_2_74_2","unstructured":"Hugging Face. 2023. Training CodeParrot from Scratch. Retrieved from https:\/\/github.com\/huggingface\/blog\/blob\/main\/codeparrot.md"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00128"},{"key":"e_1_3_2_77_2","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang et al. 2020. CodeBERT: A pre-trained model for programming and natural languages. arXiv:2002.08155. Retrieved from https:\/\/arxiv.org\/abs\/2002.08155","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_78_2","unstructured":"Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2022. InCoder: A generative model for code infilling and synthesis. arXiv:2204.05999. Retrieved from https:\/\/arxiv.org\/abs\/2204.05999"},{"key":"e_1_3_2_79_2","unstructured":"Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite Noa Nabeshima et al. 2020. The Pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.00027. Retrieved from https:\/\/arxiv.org\/abs\/2101.00027"},{"key":"e_1_3_2_80_2","first-page":"10764","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Gao Luyu","year":"2023","unstructured":"Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: Program-aided language models. In Proceedings of the International Conference on Machine Learning. PMLR, 10764\u201310799."},{"key":"e_1_3_2_81_2","unstructured":"Yunfan Gao Yun Xiong Xinyu Gao Kangxiang Jia Jinliu Pan Yuxi Bi Yi Dai Jiawei Sun and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv:2312.10997. Retrieved from https:\/\/arxiv.org\/abs\/2312.10997"},{"key":"e_1_3_2_82_2","unstructured":"Linyuan Gong Mostafa Elhoushi and Alvin Cheung. 2024. AST-T5: Structure-aware pretraining for code generation and understanding. arXiv:2401.03003. Retrieved from https:\/\/arxiv.org\/abs\/2401.03003"},{"key":"e_1_3_2_83_2","unstructured":"Alex Gu Baptiste Rozi\u00e8re Hugh Leather Armando Solar-Lezama Gabriel Synnaeve and Sida I. Wang. 2024. CRUXEval: A benchmark for code reasoning understanding and execution. arXiv:2401.03065. Retrieved from https:\/\/arxiv.org\/abs\/2401.03065"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/1836089.1836091"},{"key":"e_1_3_2_85_2","unstructured":"Suriya Gunasekar Yi Zhang Jyoti Aneja Caio C\u00e9sar Teodoro Mendes Allie Del Giorno Sivakanth Gopi Mojan Javaheripi Piero Kauffmann Gustavo de Rosa Olli Saarikivi et al. 2023. Textbooks are all you need. arXiv:2306.11644. Retrieved from https:\/\/arxiv.org\/abs\/2306.11644"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.499"},{"key":"e_1_3_2_87_2","unstructured":"Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan Alexey Svyatkovskiy Shengyu Fu et al. 2020. GraphCodeBERT: Pre-training code representations with data flow. arXiv:2009.08366. Retrieved from https:\/\/arxiv.org\/abs\/2009.08366"},{"key":"e_1_3_2_88_2","first-page":"12098","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Guo Daya","year":"2023","unstructured":"Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. 2023. LongCoder: A long-range pre-trained language model for code completion. In Proceedings of the International Conference on Machine Learning. PMLR, 12098\u201312107."},{"key":"e_1_3_2_89_2","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y. Wu Y. K. Li et al. 2024. DeepSeek-Coder: When the large language model meets programming\u2014The rise of code intelligence. arXiv:2401.14196. Retrieved from https:\/\/arxiv.org\/abs\/2401.14196"},{"key":"e_1_3_2_90_2","unstructured":"Aman Gupta Deepak Bhatt and Anubha Pandey. 2021. Transitioning from real to synthetic data: Quantifying the bias in model. arXiv:2105.04144. Retrieved from https:\/\/arxiv.org\/abs\/2105.04144"},{"key":"e_1_3_2_91_2","unstructured":"Aman Gupta Anup Shirgaonkar Angels de Luis Balaguer Bruno Silva Daniel Holstein Dawei Li Jennifer Marsman Leonardo O. Nunes Mahsa Rouzbahman Morris Sharp et al. 2024. RAG vs fine-tuning: Pipelines tradeoffs and a case study on agriculture. arXiv:2401.08406. Retrieved from https:\/\/arxiv.org\/abs\/2401.08406"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/SaTML59370.2024.00040"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580688"},{"key":"e_1_3_2_94_2","doi-asserted-by":"crossref","unstructured":"Shibo Hao Yi Gu Haodi Ma Joshua Jiahua Hong Zhen Wang Daisy Zhe Wang and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv:2305.14992. Retrieved from https:\/\/arxiv.org\/abs\/2305.14992","DOI":"10.18653\/v1\/2023.emnlp-main.507"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_96_2","unstructured":"Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Dawn Song and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv:2103.03874. Retrieved from https:\/\/arxiv.org\/abs\/2103.03874"},{"key":"e_1_3_2_97_2","unstructured":"Felipe Hoffa. 2016. GitHub on BigQuery: Analyze all the Open Source Code. Retrieved from https:\/\/cloud.google.com\/blog\/topics\/public-datasets\/github-on-bigquery-analyze-all-the-open-source-code"},{"key":"e_1_3_2_98_2","unstructured":"Jordan Hoffmann Sebastian Borgeaud Arthur Mensch Elena Buchatskaya Trevor Cai Eliza Rutherford Diego de Las Casas Lisa Anne Hendricks Johannes Welbl Aidan Clark et al. 2022. Training compute-optimal large language models. arXiv:2203.15556. Retrieved from https:\/\/arxiv.org\/abs\/2203.15556"},{"key":"e_1_3_2_99_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Holt Samuel","year":"2023","unstructured":"Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. 2023. L2MAC: Large language model automatic computer for unbounded code generation. In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_2_100_2","unstructured":"Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv:1904.09751. Retrieved from https:\/\/arxiv.org\/abs\/1904.09751"},{"key":"e_1_3_2_101_2","unstructured":"Sirui Hong Xiawu Zheng Jonathan Chen Yuheng Cheng Jinlin Wang Ceyao Zhang Zili Wang Steven Ka Shing Yau Zijuan Lin Liyang Zhou et al. 2023. MetaGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.00352. Retrieved from https:\/\/arxiv.org\/abs\/2308.00352"},{"key":"e_1_3_2_102_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. arXiv:2308.10620. Retrieved from https:\/\/arxiv.org\/abs\/2308.10620"},{"key":"e_1_3_2_103_2","first-page":"2790","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790\u20132799."},{"key":"e_1_3_2_104_2","unstructured":"Edward J. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv:2106.09685. Retrieved from https:\/\/arxiv.org\/abs\/2106.09685"},{"key":"e_1_3_2_105_2","unstructured":"Dong Huang Qingwen Bu Jie M. Zhang Michael Luck and Heming Cui. 2023. AgentCoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv:2312.13010. Retrieved from https:\/\/arxiv.org\/abs\/2312.13010"},{"key":"e_1_3_2_106_2","unstructured":"Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv:2212.10403. Retrieved from https:\/\/arxiv.org\/abs\/2212.10403"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.67"},{"key":"e_1_3_2_108_2","unstructured":"Junjie Huang Chenglong Wang Jipeng Zhang Cong Yan Haotian Cui Jeevana Priya Inala Colin Clement Nan Duan and Jianfeng Gao. 2022. Execution-based evaluation for data science code generation models. arXiv:2211.09374. Retrieved from https:\/\/arxiv.org\/abs\/2211.09374"},{"key":"e_1_3_2_109_2","unstructured":"Qiuyuan Huang Naoki Wake Bidipta Sarkar Zane Durante Ran Gong Rohan Taori Yusuke Noda Demetri Terzopoulos Noboru Kuno Ade Famoti et al. 2024. Position paper: Agent AI towards a holistic intelligence. arXiv:2403.00833. Retrieved from https:\/\/arxiv.org\/abs\/2403.00833"},{"key":"e_1_3_2_110_2","unstructured":"Binyuan Hui Jian Yang Zeyu Cui Jiaxi Yang Dayiheng Liu Lei Zhang Tianyu Liu Jiajun Zhang Bowen Yu Kai Dang et al. 2024. Qwen2.5-Coder technical report. arXiv:2409.12186. Retrieved from https:\/\/arxiv.org\/abs\/2409.12186"},{"key":"e_1_3_2_111_2","unstructured":"Hamel Husain Ho-Hsiang Wu Tiferet Gazit Miltiadis Allamanis and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv:1909.09436. Retrieved from https:\/\/arxiv.org\/abs\/1909.09436"},{"key":"e_1_3_2_112_2","unstructured":"Yoichi Ishibashi and Yoshimasa Nishimura. 2024. Self-organized agents: A LLM multi-agent framework toward ultra large-scale code generation and optimization. arXiv:2404.02183. Retrieved from https:\/\/arxiv.org\/abs\/2404.02183"},{"key":"e_1_3_2_113_2","unstructured":"Shu Ishida Gianluca Corrado George Fedoseev Hudson Yeo Lloyd Russell Jamie Shotton Jo\u00e3o F. Henriques and Anthony Hu. 2024. LangProp: A code optimization framework using language models applied to driving. arXiv:2401.10314. Retrieved from https:\/\/arxiv.org\/abs\/2401.10314"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1192"},{"key":"e_1_3_2_115_2","unstructured":"Srinivasan Iyer Xi Victoria Lin Ramakanth Pasunuru Todor Mihaylov Daniel Simig Ping Yu Kurt Shuster Tianlu Wang Qing Liu Punit Singh Koura et al. 2022. OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv:2212.12017. Retrieved from https:\/\/arxiv.org\/abs\/2212.12017"},{"key":"e_1_3_2_116_2","volume-title":"Proceedings of the 10th International Conference on Learning Representations (ICLR \u201922)","author":"Jang Joel","year":"2022","unstructured":"Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Jungkyu Choi, and Minjoon Seo. 2022. Towards continual knowledge learning of language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR \u201922). International Conference on Learning Representations."},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1121\/1.2016299"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806833"},{"key":"e_1_3_2_119_2","unstructured":"Jiaming Ji Tianyi Qiu Boyuan Chen Borong Zhang Hantao Lou Kaile Wang Yawen Duan Zhonghao He Jiayi Zhou Zhaowei Zhang et al. 2023. AI alignment: A comprehensive survey. arXiv:2310.19852. Retrieved from https:\/\/arxiv.org\/abs\/2310.19852"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3386025"},{"key":"e_1_3_2_121_2","unstructured":"Zhenlan Ji Pingchuan Ma Zongjie Li and Shuai Wang. 2023. Benchmarking and explaining large language model-based code generation: A causality-centric approach. arXiv:2310.06680. Retrieved from https:\/\/arxiv.org\/abs\/2310.06680"},{"key":"e_1_3_2_122_2","unstructured":"Juyong Jiang and Sunghun Kim. 2023. CodeUp: A Multilingual Code Generation Llama2 Model with Parameter-Efficient Instruction-Tuning. Retrieved from https:\/\/github.com\/juyongjiang\/CodeUp"},{"key":"e_1_3_2_123_2","unstructured":"Shuyang Jiang Yuhao Wang and Yu Wang. 2023. SelfEvolve: A code evolution framework via large language models. arXiv:2306.02907. Retrieved from https:\/\/arxiv.org\/abs\/2306.02907"},{"key":"e_1_3_2_124_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Jimenez Carlos E.","year":"2023","unstructured":"Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2023. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_2_125_2","unstructured":"Alexander Wettig Kilian Lieret Shunyu Yao Karthik Narasimhan Ofir Press John Yang Carlos E. Jimenez. 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. Retrieved from https:\/\/swe-agent.com\/"},{"key":"e_1_3_2_126_2","first-page":"207","volume-title":"Proceedings of the Conference on Meaning-Text Theory (MTT)","author":"Joshi Aravind","year":"2003","unstructured":"Aravind Joshi and Owen Rambow. 2003. A formalism for dependency grammar based on tree adjoining grammar. In Proceedings of the Conference on Meaning-Text Theory (MTT), 207\u2013216."},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i4.25642"},{"key":"e_1_3_2_128_2","unstructured":"Jared Kaplan Sam McCandlish Tom Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeffrey Wu and Dario Amodei. 2020. Scaling laws for neural language models. arXiv:2001.08361. Retrieved from https:\/\/arxiv.org\/abs\/2001.08361"},{"key":"e_1_3_2_129_2","unstructured":"Mohammad Abdullah Matin Khan M. Saiful Bari Xuan Long Do Weishi Wang Md Rizwan Parvez and Shafiq Joty. 2023. xCodeEval: A large scale multilingual multitask benchmark for code understanding generation translation and retrieval. arXiv:2303.03004. Retrieved from https:\/\/arxiv.org\/abs\/2303.03004"},{"key":"e_1_3_2_130_2","unstructured":"Dahyun Kim Yungi Kim Wonho Song Hyeonwoo Kim Yunsu Kim Sanghoon Kim and Chanjun Park. 2024. sDPO: Don\u2019t use your data all at once. arXiv:2403.19270. Retrieved from https:\/\/arxiv.org\/abs\/2403.19270"},{"key":"e_1_3_2_131_2","doi-asserted-by":"crossref","unstructured":"Dahyun Kim Chanjun Park Sanghoon Kim Wonsung Lee Wonho Song Yunsu Kim Hyeonwoo Kim Yungi Kim Hyeonju Lee Jihoo Kim et al. 2023. SOLAR 10.7B: Scaling large language models with simple yet effective depth up-scaling. arXiv:2312.15166. Retrieved from https:\/\/arxiv.org\/abs\/2312.15166","DOI":"10.18653\/v1\/2024.naacl-industry.3"},{"key":"e_1_3_2_132_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_3_2_133_2","unstructured":"Denis Kocetkov Raymond Li L. I. Jia Chenghao Mou Yacine Jernite Margaret Mitchell Carlos Mu\u00f1oz Ferrandis Sean Hughes Thomas Wolf Dzmitry Bahdanau et al. 2022. The Stack: 3 TB of permissively licensed source code. arXiv:2211.15533. Retrieved from https:\/\/arxiv.org\/abs\/2211.15533"},{"key":"e_1_3_2_134_2","article-title":"OpenAssistant Conversations\u2014Democratizing large language model alignment","volume":"36","author":"K\u00f6pf Andreas","year":"2024","unstructured":"Andreas K\u00f6pf, Yannic Kilcher, Dimitri von R\u00fctte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich\u00e1rd Nagyfi, et al. 2024. OpenAssistant Conversations\u2014Democratizing large language model alignment. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_135_2","unstructured":"Bonan Kou Shengmai Chen Zhijie Wang Lei Ma and Tianyi Zhang. 2023. Is model attention aligned with human attention? An empirical study on large language models for code generation. arXiv:2306.01220. Retrieved from https:\/\/arxiv.org\/abs\/2306.01220"},{"key":"e_1_3_2_136_2","unstructured":"Marie-Anne Lachaux Baptiste Roziere Lowik Chanussot and Guillaume Lample. 2020. Unsupervised translation of programming languages. arXiv:2006.03511. Retrieved from https:\/\/arxiv.org\/abs\/2006.03511"},{"key":"e_1_3_2_137_2","first-page":"18319","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Lai Yuhang","year":"2023","unstructured":"Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning. PMLR, 18319\u201318345."},{"key":"e_1_3_2_138_2","first-page":"31809","article-title":"The BigScience ROOTS Corpus: A 1.6 TB composite multilingual dataset","volume":"35","author":"Lauren\u00e7on Hugo","year":"2022","unstructured":"Hugo Lauren\u00e7on, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo Gonz\u00e1lez Ponferrada, Huu Nguyen, et al. 2022. The BigScience ROOTS Corpus: A 1.6 TB composite multilingual dataset. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 31809\u201331826.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_139_2","unstructured":"Moritz Laurer. 2024. Synthetic Data: Save Money Time and Carbon with Open Source. Retrieved from https:\/\/huggingface.co\/blog\/synthetic-data-save-costs"},{"key":"e_1_3_2_140_2","first-page":"21314","article-title":"CodeRL: Mastering code generation through pretrained models and deep reinforcement learning","volume":"35","author":"Le Hung","year":"2022","unstructured":"Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 21314\u201321328.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_141_2","unstructured":"Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ili\u0107 Daniel Hesslow Roman Castagn\u00e9 Alexandra Sasha Luccioni Fran\u00e7ois Yvon Matthias Gall\u00e9 et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https:\/\/arxiv.org\/abs\/2211.05100"},{"key":"e_1_3_2_142_2","unstructured":"Harrison Lee Samrat Phatale Hassan Mansoor Kellie Lu Thomas Mesnard Colton Bishop Victor Carbune and Abhinav Rastogi. 2023. RLAIF: Scaling reinforcement learning from human feedback with AI feedback. arXiv:2309.00267. Retrieved from https:\/\/arxiv.org\/abs\/2309.00267"},{"key":"e_1_3_2_143_2","doi-asserted-by":"crossref","unstructured":"Brian Lester Rami Al-Rfou and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv:2104.08691. Retrieved from https:\/\/arxiv.org\/abs\/2104.08691","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_3_2_144_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 33, 9459\u20139474.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_145_2","unstructured":"Jia Li Yunfei Zhao Yongmin Li Ge Li and Zhi Jin. 2023. Towards enhancing in-context learning for code generation. arXiv:2303.17780. Retrieved from https:\/\/arxiv.org\/abs\/2303.17780"},{"key":"e_1_3_2_146_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.04.001"},{"key":"e_1_3_2_147_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim et al. 2023. StarCoder: May the source be with you! arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_2_148_2","unstructured":"Xuechen Li Tianyi Zhang Yann Dubois Rohan Taori Ishaan Gulrajani Carlos Guestrin Percy Liang and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-Following Models. Retrieved from https:\/\/github.com\/tatsu-lab\/alpaca_eval"},{"key":"e_1_3_2_149_2","unstructured":"Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv:2101.00190. Retrieved from https:\/\/arxiv.org\/abs\/2101.00190"},{"key":"e_1_3_2_150_2","unstructured":"Yuanzhi Li S\u00e9bastien Bubeck Ronen Eldan Allie Del Giorno Suriya Gunasekar and Yin Tat Lee. 2023. Textbooks are all you need II: Phi-1.5 technical report. arXiv:2309.05463. Retrieved from https:\/\/arxiv.org\/abs\/2309.05463"},{"key":"e_1_3_2_151_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_2_152_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510217"},{"key":"e_1_3_2_153_2","unstructured":"Vladislav Lialin Vijeta Deshpande and Anna Rumshisky. 2023. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv:2303.15647. Retrieved from https:\/\/arxiv.org\/abs\/2303.15647"},{"key":"e_1_3_2_154_2","unstructured":"Percy Liang Rishi Bommasani Tony Lee Dimitris Tsipras Dilara Soylu Michihiro Yasunaga Yian Zhang Deepak Narayanan Yuhuai Wu Ananya Kumar et al. 2022. Holistic evaluation of language models. arXiv:2211.09110. Retrieved from https:\/\/arxiv.org\/abs\/2211.09110"},{"key":"e_1_3_2_155_2","doi-asserted-by":"publisher","DOI":"10.1145\/3571884.3604316"},{"key":"e_1_3_2_156_2","article-title":"ROUGE: A package for automatic evaluation of summaries","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. . Association for Computational Linguistics, Barcelona, Spain, 74\u201381.","journal-title":"Text Summarization Branches Out"},{"key":"e_1_3_2_157_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2022.10.001"},{"key":"e_1_3_2_158_2","article-title":"Mining internet-scale software repositories","volume":"20","author":"Linstead Erik","year":"2007","unstructured":"Erik Linstead, Paul Rigor, Sushil Bajracharya, Cristina Lopes, and Pierre Baldi. 2007. Mining internet-scale software repositories. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 20.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_159_2","unstructured":"Aixin Liu Bei Feng Bin Wang Bingxuan Wang Bo Liu Chenggang Zhao Chengqi Dengr Chong Ruan Damai Dai Daya Guo et al. 2024. DeepSeek-V2: A strong economical and efficient mixture-of-experts language model. arXiv:2405.04434. Retrieved from https:\/\/arxiv.org\/abs\/2405.04434"},{"key":"e_1_3_2_160_2","unstructured":"Bingchang Liu Chaoyu Chen Cong Liao Zi Gong Huan Wang Zhichao Lei Ming Liang Dajun Chen Min Shen Hailian Zhou et al. 2023. MFTCoder: Boosting code LLMs with multitask fine-tuning. arXiv:2311.02303. Retrieved from https:\/\/arxiv.org\/abs\/2311.02303"},{"key":"e_1_3_2_161_2","first-page":"1950","article-title":"Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning","volume":"35","author":"Liu Haokun","year":"2022","unstructured":"Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A. Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 1950\u20131965.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_162_2","article-title":"Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation","volume":"36","author":"Liu Jiawei","year":"2024","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_163_2","unstructured":"Jiate Liu Yiqin Zhu Kaiwen Xiao Qiang Fu Xiao Han Wei Yang and Deheng Ye. 2023. RLTF: Reinforcement learning from unit test feedback. arXiv:2307.04349. Retrieved from https:\/\/arxiv.org\/abs\/2307.04349"},{"key":"e_1_3_2_164_2","doi-asserted-by":"crossref","unstructured":"Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi and Graham Neubig. 2023. Pre-train prompt and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55 9 (2023) 1\u201335.","DOI":"10.1145\/3560815"},{"key":"e_1_3_2_165_2","unstructured":"Ruibo Liu Jerry Wei Fangyu Liu Chenglei Si Yanzhe Zhang Jinmeng Rao Steven Zheng Daiyi Peng Diyi Yang Denny Zhou et al. 2024. Best practices and lessons learned on synthetic data for language models. arXiv:2404.07503. Retrieved from https:\/\/arxiv.org\/abs\/2404.07503"},{"key":"e_1_3_2_166_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Liu Shangqing","year":"2020","unstructured":"Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2020. Retrieval-augmented generation for code summarization via hybrid GNN. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_167_2","first-page":"2368","article-title":"Uncovering and quantifying social biases in code generation","volume":"36","author":"Liu Yan","year":"2023","unstructured":"Yan Liu, Xiaokang Chen, Yan Gao, Zhe Su, Fengji Zhang, Daoguang Zan, Jian-Guang Lou, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Uncovering and quantifying social biases in code generation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36, 2368\u20132380.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_168_2","doi-asserted-by":"crossref","unstructured":"Yue Liu Chakkrit Tantithamthavorn Li Li and Yepang Liu. 2022. Deep learning for android malware defenses: A systematic literature review. ACM Computing Surveys 55 8 (2022) 1\u201336.","DOI":"10.1145\/3544968"},{"key":"e_1_3_2_169_2","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei et al. 2024. StarCoder 2 and The Stack v2: The next generation. arXiv:2402.19173. Retrieved from https:\/\/arxiv.org\/abs\/2402.19173"},{"key":"e_1_3_2_170_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.431"},{"key":"e_1_3_2_171_2","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin Clement Dawn Drain Daxin Jiang Duyu Tang et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arXiv:2102.04664. Retrieved from https:\/\/arxiv.org\/abs\/2102.04664"},{"key":"e_1_3_2_172_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Luo Ziyang","year":"2023","unstructured":"Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering code large language models with Evol-Instruct. In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_2_173_2","unstructured":"Michael R. Lyu Baishakhi Ray Abhik Roychoudhury Shin Hwei Tan and Patanamon Thongtanunam. 2024. Automatic programming: Large language models and beyond. arXiv:2405.02213. Retrieved from https:\/\/arxiv.org\/abs\/2405.02213"},{"key":"e_1_3_2_174_2","unstructured":"Wei Ma Mengjie Zhao Xiaofei Xie Qiang Hu Shangqing Liu Jie Zhang Wenhan Wang and Yang Liu. 2022. Are code pre-trained models powerful to learn code syntax and semantics? arXiv:2212.10017. Retrieved from https:\/\/arxiv.org\/abs\/2212.10017"},{"key":"e_1_3_2_175_2","article-title":"Self-refine: Iterative refinement with self-feedback","volume":"36","author":"Madaan Aman","year":"2024","unstructured":"Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_176_2","article-title":"An overview of Bard: An early experiment with generative AI","volume":"2","author":"Manyika James","year":"2023","unstructured":"James Manyika and Sissie Hsiao. 2023. An overview of Bard: An early experiment with generative AI. Google Static Documents 2 (2023).","journal-title":"Google Static Documents"},{"key":"e_1_3_2_177_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00073"},{"key":"e_1_3_2_178_2","first-page":"462","article-title":"Generating training data with language models: Towards zero-shot language understanding","volume":"35","author":"Meng Yu","year":"2022","unstructured":"Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 462\u2013477.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_179_2","unstructured":"Meta. 2024. Introducing Meta Llama 3: The Most Capable Openly Available LLM to Date. Retrieved from https:\/\/ai.meta.com\/blog\/meta-llama-3\/"},{"key":"e_1_3_2_180_2","unstructured":"MistralAI. 2024. Codestral. Retrieved from https:\/\/mistral.ai\/news\/codestral\/"},{"key":"e_1_3_2_181_2","unstructured":"S\u00e9bastien Bubeck Mojan Javaheripi. 2023. Phi-2: The Surprising Power of Small Language Models. Retrieved from https:\/\/www.microsoft.com\/en-us\/research\/blog\/phi-2-the-surprising-power-of-small-language-models"},{"key":"e_1_3_2_182_2","unstructured":"Lili Mou Ge Li Zhi Jin Lu Zhang and Tao Wang. 2014. TBCNN: A tree-based convolutional neural network for programming language processing. arXiv:1409.5718. Retrieved from https:\/\/arxiv.org\/abs\/1409.5718"},{"key":"e_1_3_2_183_2","doi-asserted-by":"publisher","DOI":"10.1145\/3634737.3661134"},{"key":"e_1_3_2_184_2","unstructured":"Spyridon Mouselinos Mateusz Malinowski and Henryk Michalewski. 2022. A simple yet effective approach to finding biases in code generation. arXiv:2211.00609. Retrieved from https:\/\/arxiv.org\/abs\/2211.00609"},{"key":"e_1_3_2_185_2","unstructured":"Hussein Mozannar Gagan Bansal Adam Fourney and Eric Horvitz. 2022. Reading between the lines: Modeling user behavior and costs in AI-assisted programming. arXiv:2210.14306. Retrieved from https:\/\/arxiv.org\/abs\/2210.14306"},{"key":"e_1_3_2_186_2","unstructured":"Niklas Muennighoff Qian Liu Armel Zebaze Qinkai Zheng Binyuan Hui Terry Yue Zhuo Swayam Singh Xiangru Tang Leandro Von Werra and Shayne Longpre. 2023. OctoPack: Instruction tuning code large language models. arXiv:2308.07124. Retrieved from https:\/\/arxiv.org\/abs\/2308.07124"},{"key":"e_1_3_2_187_2","unstructured":"King Han Naman Jain Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar-Lezama Koushik Sen and Ion Stoica. 2024. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv:2403.07974. Retrieved from https:\/\/arxiv.org\/abs\/2403.07974"},{"key":"e_1_3_2_188_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2015.48"},{"key":"e_1_3_2_189_2","first-page":"26106","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Ni Ansong","year":"2023","unstructured":"Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. LEVER: Learning to verify language-to-code generation with execution. In Proceedings of the International Conference on Machine Learning. PMLR, 26106\u201326128."},{"key":"e_1_3_2_190_2","unstructured":"Ansong Ni Pengcheng Yin Yilun Zhao Martin Riddell Troy Feng Rui Shen Stephen Yin Ye Liu Semih Yavuz Caiming Xiong et al. 2023. L2CEval: Evaluating language-to-code generation capabilities of large language models. arXiv:2309.17446. Retrieved from https:\/\/arxiv.org\/abs\/2309.17446"},{"key":"e_1_3_2_191_2","unstructured":"Erik Nijkamp Hiroaki Hayashi Caiming Xiong Silvio Savarese and Yingbo Zhou. 2023. CodeGen2: Lessons for training LLMs on programming and natural languages. arXiv:2305.02309. Retrieved from https:\/\/arxiv.org\/abs\/2305.02309"},{"key":"e_1_3_2_192_2","unstructured":"Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from https:\/\/arxiv.org\/abs\/2203.13474"},{"key":"e_1_3_2_193_2","unstructured":"Changan Niu Chuanyi Li Bin Luo and Vincent Ng. 2022. Deep learning meets software engineering: A survey on pre-trained models of source code. arXiv:2205.11739. Retrieved from https:\/\/arxiv.org\/abs\/2205.11739"},{"key":"e_1_3_2_194_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Olausson Theo X.","year":"2023","unstructured":"Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Is self-repair a silver bullet for code generation? In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_2_195_2","unstructured":"OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. Retrieved from https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_3_2_196_2","unstructured":"OpenAI. 2024. Hello GPT-4o. Retrieved from https:\/\/openai.com\/index\/hello-gpt-4o\/"},{"key":"e_1_3_2_197_2","unstructured":"OpenAI. 2024. New Models and Developer Products Announced at DevDay. Retrieved from https:\/\/openai.com\/index\/new-models-and-developer-products-announced-at-devday\/"},{"key":"e_1_3_2_198_2","unstructured":"OpenDevin. 2024. OpenDevin: Code Less Make More. Retrieved from https:\/\/github.com\/OpenDevin\/OpenDevin"},{"key":"e_1_3_2_199_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 27730\u201327744.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_200_2","unstructured":"Oded Ovadia Menachem Brief Moshik Mishaeli and Oren Elisha. 2023. Fine-tuning or retrieval? Comparing knowledge injection in LLMs. arXiv:2312.05934. Retrieved from https:\/\/arxiv.org\/abs\/2312.05934"},{"key":"e_1_3_2_201_2","unstructured":"David N. Palacio Alejandro Velasco Daniel Rodriguez-Cardenas Kevin Moran and Denys Poshyvanyk. 2023. Evaluating and explaining large language models for code using syntactic structures. arXiv:2308.03873. Retrieved from https:\/\/arxiv.org\/abs\/2308.03873"},{"key":"e_1_3_2_202_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_2_203_2","unstructured":"Nikhil Parasaram Huijie Yan Boyu Yang Zineb Flahy Abriele Qudsi Damian Ziaber Earl Barr and Sergey Mechtaev. 2024. The fact selection problem in LLM-based program repair. arXiv:2404.05520. Retrieved from https:\/\/arxiv.org\/abs\/2404.05520"},{"key":"e_1_3_2_204_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.232"},{"key":"e_1_3_2_205_2","unstructured":"Arkil Patel Siva Reddy Dzmitry Bahdanau and Pradeep Dasigi. 2023. Evaluating in-context learning of libraries for code generation. arXiv:2311.09635. Retrieved from https:\/\/arxiv.org\/abs\/2311.09635"},{"key":"e_1_3_2_206_2","unstructured":"Indraneil Paul Jun Luo Goran Glava\u0161 and Iryna Gurevych. 2024. IRCoder: Intermediate representations make language models robust multilingual code generators. arXiv:2403.03894. Retrieved from https:\/\/arxiv.org\/abs\/2403.03894"},{"key":"e_1_3_2_207_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00056"},{"key":"e_1_3_2_208_2","unstructured":"Nikhil Pinnaparaju Reshinth Adithyan Duy Phung Jonathan Tow James Baicoianu Ashish Datta Maksym Zhuravinskyi Dakota Mahan Marco Bellagente Carlos Riquelme et al. 2024. Stable code technical report. arXiv:2404.01226. Retrieved from https:\/\/arxiv.org\/abs\/2404.01226"},{"key":"e_1_3_2_209_2","unstructured":"Ofir Press Noah A. Smith and Mike Lewis. 2021. Train short test long: Attention with linear biases enables input length extrapolation. arXiv:2108.12409. Retrieved from https:\/\/arxiv.org\/abs\/2108.12409"},{"key":"e_1_3_2_210_2","unstructured":"Xiangyu Qi Yi Zeng Tinghao Xie Pin-Yu Chen Ruoxi Jia Prateek Mittal and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety even when users do not intend to! arXiv:2310.03693. Retrieved from https:\/\/arxiv.org\/abs\/2310.03693"},{"key":"e_1_3_2_211_2","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans and Ilya Sutskever. 2018. Improving language understanding by generative pre-training (2018). Retrieved from https:\/\/www.semanticscholar.org\/paper\/Improving-Language-Understanding-by-Generative-Radford-Narasimhan\/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035"},{"issue":"8","key":"e_1_3_2_212_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_213_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2012.6405296"},{"key":"e_1_3_2_214_2","article-title":"Direct preference optimization: Your language model is secretly a reward model","volume":"36","author":"Rafailov Rafael","year":"2024","unstructured":"Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"issue":"140","key":"e_1_3_2_215_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1\u201367.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_216_2","unstructured":"Nitarshan Rajkumar Raymond Li and Dzmitry Bahdanau. 2022. Evaluating the text-to-SQL capabilities of large language models. arXiv:2204.00498. Retrieved from https:\/\/arxiv.org\/abs\/2204.00498"},{"key":"e_1_3_2_217_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2803055"},{"key":"e_1_3_2_218_2","unstructured":"Machel Reid Nikolay Savinov Denis Teplyashin Dmitry Lepikhin Timothy Lillicrap Jean-baptiste Alayrac Radu Soricut Angeliki Lazaridou Orhan Firat Julian Schrittwieser et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530. Retrieved from https:\/\/arxiv.org\/abs\/2403.05530"},{"key":"e_1_3_2_219_2","unstructured":"Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan Ming Zhou Ambrosio Blanco and Shuai Ma. 2020. CodeBLEU: A method for automatic evaluation of code synthesis. arXiv:2009.10297. Retrieved from https:\/\/arxiv.org\/abs\/2009.10297"},{"key":"e_1_3_2_220_2","unstructured":"Replit. 2016. Idea to Software Fast. Retrieved from https:\/\/replit.com"},{"key":"e_1_3_2_221_2","unstructured":"Replit. 2023. replit-code-v1-3b. Retrieved from https:\/\/huggingface.co\/replit\/replit-code-v1-3b"},{"key":"e_1_3_2_222_2","unstructured":"Tal Ridnik Dedy Kredo and Itamar Friedman. 2024. Code generation with AlphaCodium: From prompt engineering to flow engineering. arXiv:2401.08500. Retrieved from https:\/\/arxiv.org\/abs\/2401.08500"},{"key":"e_1_3_2_223_2","unstructured":"Nick Roshdieh. 2023. Evol-Instruct-Code-80k. Retrieved from https:\/\/huggingface.co\/datasets\/nickrosh\/Evol-Instruct-Code-80k-v1"},{"key":"e_1_3_2_224_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581641.3584037"},{"key":"e_1_3_2_225_2","unstructured":"Baptiste Roziere Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin et al. 2023. Code Llama: Open foundation models for code. arXiv:2308.12950. Retrieved from https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_2_226_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC58863.2023.10363447"},{"key":"e_1_3_2_227_2","volume-title":"Proceedings of the 10th International Conference on Learning Representations (ICLR \u201922)","author":"Sanh Victor","year":"2022","unstructured":"Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In Proceedings of the 10th International Conference on Learning Representations (ICLR \u201922)."},{"key":"e_1_3_2_228_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. Retrieved from https:\/\/arxiv.org\/abs\/1707.06347"},{"key":"e_1_3_2_229_2","first-page":"1559","volume-title":"Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921)","author":"Schuster Roei","year":"2021","unstructured":"Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In Proceedings of the 30th USENIX Security Symposium (USENIX Security \u201921), 1559\u20131575."},{"key":"e_1_3_2_230_2","unstructured":"Peter Shaw Jakob Uszkoreit and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv:1803.02155. Retrieved from https:\/\/arxiv.org\/abs\/1803.02155"},{"key":"e_1_3_2_231_2","unstructured":"Noam Shazeer Azalia Mirhoseini Krzysztof Maziarz Andy Davis Quoc Le Geoffrey Hinton and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv:1701.06538. Retrieved from https:\/\/arxiv.org\/abs\/1701.06538"},{"key":"e_1_3_2_232_2","unstructured":"Bo Shen Jiaxin Zhang Taihong Chen Daoguang Zan Bing Geng An Fu Muhan Zeng Ailun Yu Jichuan Ji Jingyang Zhao et al. 2023. PanGu-Coder2: Boosting large language models for code with ranking feedback. arXiv:2307.14936. Retrieved from https:\/\/arxiv.org\/abs\/2307.14936"},{"key":"e_1_3_2_233_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639475.3640097"},{"key":"e_1_3_2_234_2","article-title":"Reflexion: Language agents with verbal reinforcement learning","volume":"36","author":"Shinn Noah","year":"2024","unstructured":"Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_235_2","doi-asserted-by":"crossref","unstructured":"Atsushi Shirafuji Yusuke Oda Jun Suzuki Makoto Morishita and Yutaka Watanobe. 2023. Refactoring programs using large language models with few-shot examples. arXiv:2311.11690. Retrieved from https:\/\/arxiv.org\/abs\/2311.11690","DOI":"10.1109\/APSEC60848.2023.00025"},{"key":"e_1_3_2_236_2","unstructured":"Parshin Shojaee Aneesh Jain Sindhu Tipirneni and Chandan K. Reddy. 2023. Execution-based code generation using deep reinforcement learning. arXiv:2301.13816. Retrieved from https:\/\/arxiv.org\/abs\/2301.13816"},{"key":"e_1_3_2_237_2","unstructured":"Disha Shrivastava Denis Kocetkov Harm de Vries Dzmitry Bahdanau and Torsten Scholak. 2023. RepoFusion: Training code models to understand your repository. arXiv:2306.10998. Retrieved from https:\/\/arxiv.org\/abs\/2306.10998"},{"key":"e_1_3_2_238_2","first-page":"31693","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Shrivastava Disha","year":"2023","unstructured":"Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In Proceedings of the International Conference on Machine Learning. PMLR, 31693\u201331715."},{"key":"e_1_3_2_239_2","doi-asserted-by":"crossref","unstructured":"Mukul Singh Jos\u00e9 Cambronero Sumit Gulwani Vu Le Carina Negreanu and Gust Verbruggen. 2023. CodeFusion: A pre-trained diffusion model for code generation. arXiv:2310.17680. Retrieved from https:\/\/arxiv.org\/abs\/2310.17680","DOI":"10.18653\/v1\/2023.emnlp-main.716"},{"key":"e_1_3_2_240_2","unstructured":"Hongjin Su Shuyang Jiang Yuhang Lai Haoyuan Wu Boao Shi Che Liu Qian Liu and Tao Yu. 2024. ARKS: Active retrieval in knowledge soup for code generation. arXiv:2402.12317. Retrieved from https:\/\/arxiv.org\/abs\/2402.12317"},{"key":"e_1_3_2_241_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2023.127063"},{"key":"e_1_3_2_242_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_3_2_243_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR)","author":"Szafraniec Marc","year":"2022","unstructured":"Marc Szafraniec, Baptiste Roziere, Hugh Leather, Francois Charton, Patrick Labatut, and Gabriel Synnaeve. 2022. Code translation with compiler representations. In Proceedings of the 11th International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_244_2","unstructured":"TabNine. 2018. AI Code Completions. Retrieved from https:\/\/github.com\/codota\/TabNine"},{"key":"e_1_3_2_245_2","unstructured":"Rohan Taori Ishaan Gulrajani Tianyi Zhang Yann Dubois Xuechen Li Carlos Guestrin Percy Liang and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-Following LLaMA Model. Retrieved from https:\/\/github.com\/tatsu-lab\/stanford_alpaca"},{"key":"e_1_3_2_246_2","unstructured":"Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju Shreya Pathak Laurent Sifre Morgane Rivi\u00e8re Mihir Sanjay Kale Juliette Love et al. 2024. Gemma: Open models based on Gemini research and technology. arXiv:2403.08295. Retrieved from https:\/\/arxiv.org\/abs\/2403.08295"},{"key":"e_1_3_2_247_2","unstructured":"Qwen Team. 2024. Code with CodeQwen1.5. Retrieved from https:\/\/qwenlm.github.io\/blog\/codeqwen1.5"},{"key":"e_1_3_2_248_2","doi-asserted-by":"publisher","DOI":"10.23919\/DATE56975.2023.10137086"},{"key":"e_1_3_2_249_2","unstructured":"theblackcat102. 2023. The Evolved Code Alpaca Dataset. Retrieved from https:\/\/huggingface.co\/datasets\/theblackcat102\/evol-codealpaca-v1"},{"key":"e_1_3_2_250_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_2_251_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_252_2","volume-title":"Natural Language Processing with Transformers","author":"Tunstall Lewis","year":"2022","unstructured":"Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. 2022. Natural Language Processing with Transformers. O\u2019Reilly Media, Inc."},{"key":"e_1_3_2_253_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_2_254_2","first-page":"34793","volume-title":"Proceedings of the International Conference on Machine Learning. PMLR","author":"Van Breugel Boris","year":"2023","unstructured":"Boris Van Breugel, Zhaozhi Qian, and Mihaela Van Der Schaar. 2023. Synthetic data, real errors: How (not) to publish and use synthetic data. In Proceedings of the International Conference on Machine Learning. PMLR, 34793\u201334808."},{"key":"e_1_3_2_255_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_256_2","unstructured":"Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. Retrieved from https:\/\/github.com\/kingoflolz\/mesh-transformer-jax"},{"key":"e_1_3_2_257_2","doi-asserted-by":"crossref","unstructured":"Chong Wang Jian Zhang Yebo Feng Tianlin Li Weisong Sun Yang Liu and Xin Peng. 2024. Teaching code LLMs to use autocompletion tools in repository-level code generation. arXiv:2401.06391. Retrieved from https:\/\/arxiv.org\/abs\/2401.06391","DOI":"10.1145\/3714462"},{"key":"e_1_3_2_258_2","unstructured":"Fan Wang Juyong Jiang Chansung Park Sunghun Kim and Jing Tang. 2024. KaSA: Knowledge-aware singular-value adaptation of large language models. arXiv:2412.06071. Retrieved from https:\/\/arxiv.org\/abs\/2412.06071"},{"key":"e_1_3_2_259_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_2_260_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-40231-1"},{"key":"e_1_3_2_261_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3173346"},{"key":"e_1_3_2_262_2","unstructured":"Shiqi Wang Li Zheng Haifeng Qian Chenghao Yang Zijian Wang Varun Kumar Mingyue Shang Samson Tan Baishakhi Ray Parminder Bhatia et al. 2022. ReCode: Robustness evaluation of code generation models. arXiv:2212.10264. Retrieved from https:\/\/arxiv.org\/abs\/2212.10264"},{"key":"e_1_3_2_263_2","unstructured":"Song Wang Yaochen Zhu Haochen Liu Zaiyi Zheng Chen Chen and Jundong Li. 2023. Knowledge editing for large language models: A survey. arXiv:2310.16218. Retrieved from https:\/\/arxiv.org\/abs\/2310.16218"},{"key":"e_1_3_2_264_2","unstructured":"Xingyao Wang Yangyi Chen Lifan Yuan Yizhe Zhang Yunzhu Li Hao Peng and Heng Ji. 2024. Executable code actions elicit better LLM agents. arXiv:2402.01030. Retrieved from https:\/\/arxiv.org\/abs\/2402.01030"},{"key":"e_1_3_2_265_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-acl.2"},{"key":"e_1_3_2_266_2","unstructured":"Xuezhi Wang Jason Wei Dale Schuurmans Quoc Le Ed Chi Sharan Narang Aakanksha Chowdhery and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Retrieved from https:\/\/arxiv.org\/abs\/2203.11171"},{"key":"e_1_3_2_267_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"e_1_3_2_268_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.68"},{"key":"e_1_3_2_269_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17650"},{"key":"e_1_3_2_270_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_271_2","unstructured":"Yufei Wang Wanjun Zhong Liangyou Li Fei Mi Xingshan Zeng Wenyong Huang Lifeng Shang Xin Jiang and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv:2307.12966. Retrieved from https:\/\/arxiv.org\/abs\/2307.12966"},{"key":"e_1_3_2_272_2","unstructured":"Zhiruo Wang Shuyan Zhou Daniel Fried and Graham Neubig. 2022. Execution-based evaluation for open-domain code generation. arXiv:2212.10481. Retrieved from https:\/\/arxiv.org\/abs\/2212.10481"},{"key":"e_1_3_2_273_2","unstructured":"Jason Wei Maarten Bosma Vincent Y. Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. arXiv:2109.01652. Retrieved from https:\/\/arxiv.org\/abs\/2109.01652"},{"key":"e_1_3_2_274_2","unstructured":"Jason Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph Sebastian Borgeaud Dani Yogatama Maarten Bosma Denny Zhou Donald Metzler et al. 2022. Emergent abilities of large language models. arXiv:2206.07682. Retrieved from https:\/\/arxiv.org\/abs\/2206.07682"},{"key":"e_1_3_2_275_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_276_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616302"},{"key":"e_1_3_2_277_2","unstructured":"Yuxiang Wei Zhe Wang Jiawei Liu Yifeng Ding and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv:2312.02120. Retrieved from https:\/\/arxiv.org\/abs\/2312.02120"},{"key":"e_1_3_2_278_2","unstructured":"Lilian Weng. 2023. LLM-Powered Autonomous Agents. Retrieved June 2023 from https:\/\/lilianweng.github.io\/posts\/2023-06-23-agent\/"},{"key":"e_1_3_2_279_2","unstructured":"Alexander Wettig Aatmik Gupta Saumya Malik and Danqi Chen. 2024. QuRating: Selecting high-quality data for training language models. arXiv:2402.09739. Retrieved from https:\/\/arxiv.org\/abs\/2402.09739"},{"key":"e_1_3_2_280_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00366"},{"key":"e_1_3_2_281_2","unstructured":"Di Wu Wasi Uddin Ahmad Dejiao Zhang Murali Krishna Ramanathan and Xiaofei Ma. 2024. Repoformer: Selective retrieval for repository-level code completion. arXiv:2403.10059. Retrieved from https:\/\/arxiv.org\/abs\/2403.10059"},{"key":"e_1_3_2_282_2","unstructured":"Qingyun Wu Gagan Bansal Jieyu Zhang Yiran Wu Shaokun Zhang Erkang Zhu Beibin Li Li Jiang Xiaoyun Zhang and Chi Wang. 2023. AutoGen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv:2308.08155. Retrieved from https:\/\/arxiv.org\/abs\/2308.08155"},{"key":"e_1_3_2_283_2","unstructured":"Zhiheng Xi Wenxiang Chen Xin Guo Wei He Yiwen Ding Boyang Hong Ming Zhang Junzhe Wang Senjie Jin Enyu Zhou et al. 2023. The rise and potential of large language model based agents: A survey. arXiv:2309.07864. Retrieved from https:\/\/arxiv.org\/abs\/2309.07864"},{"key":"e_1_3_2_284_2","unstructured":"Rui Xie Zhengran Zeng Zhuohao Yu Chang Gao Shikun Zhang and Wei Ye. 2024. CodeShell technical report. arXiv:2403.15747. Retrieved from https:\/\/arxiv.org\/abs\/2403.15747"},{"key":"e_1_3_2_285_2","first-page":"34201","article-title":"Data selection for language models via importance resampling","volume":"36","author":"Xie Sang Michael","year":"2023","unstructured":"Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S. Liang. 2023. Data selection for language models via importance resampling. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36, 34201\u201334227.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_286_2","unstructured":"Xingyao Wang Bowen Li and Graham Neubig. 2024. Introducing OpenDevin CodeAct 1.0 a New State-of-the-Art in Coding Agents. Retrieved from https:\/\/www.cognition.ai\/introducing-devin"},{"key":"e_1_3_2_287_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.45"},{"key":"e_1_3_2_288_2","unstructured":"Can Xu Qingfeng Sun Kai Zheng Xiubo Geng Pu Zhao Jiazhan Feng Chongyang Tao and Daxin Jiang. 2023. WizardLM: Empowering large language models to follow complex instructions. arXiv:2304.12244. Retrieved from https:\/\/arxiv.org\/abs\/2304.12244"},{"key":"e_1_3_2_289_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_2_290_2","unstructured":"Junjielong Xu Ying Fu Shin Hwei Tan and Pinjia He. 2024. Aligning LLMs for FL-free program repair. arXiv:2404.08877. Retrieved from https:\/\/arxiv.org\/abs\/2404.08877"},{"key":"e_1_3_2_291_2","unstructured":"Weiwei Xu Kai Gao Hao He and Minghui Zhou. 2024. A first look at license compliance capability of LLMs in code generation. arXiv:2408.02487. Retrieved from https:\/\/arxiv.org\/abs\/2408.02487"},{"key":"e_1_3_2_292_2","unstructured":"Zhou Yang Zhensu Sun Terry Zhuo Yue Premkumar Devanbu and David Lo. 2024. Robustness security privacy explainability efficiency and usability of large language models for code. arXiv:2403.07506. Retrieved from https:\/\/arxiv.org\/abs\/2403.07506"},{"key":"e_1_3_2_293_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639074"},{"key":"e_1_3_2_294_2","article-title":"Tree of thoughts: Deliberate problem solving with large language models","volume":"36","author":"Yao Shunyu","year":"2024","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_295_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_296_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196408"},{"key":"e_1_3_2_297_2","unstructured":"Kang Min Yoo Jaegeun Han Sookyo In Heewon Jeon Jisu Jeong Jaewook Kang Hyunwook Kim Kyung-Min Kim Munhyong Kim Sungju Kim et al. 2024. HyperCLOVA X technical report. arXiv:2404.01954. Retrieved from https:\/\/arxiv.org\/abs\/2404.01954"},{"key":"e_1_3_2_298_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623316"},{"key":"e_1_3_2_299_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1425"},{"key":"e_1_3_2_300_2","unstructured":"Zhaojian Yu Xin Zhang Ning Shang Yangyu Huang Can Xu Yishujie Zhao Wenxiang Hu and Qiufeng Yin. 2023. WaveCoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv:2312.14187. Retrieved from https:\/\/arxiv.org\/abs\/2312.14187"},{"key":"e_1_3_2_301_2","unstructured":"Youliang Yuan Wenxiang Jiao Wenxuan Wang Jen-tse Huang Pinjia He Shuming Shi and Zhaopeng Tu. 2023. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. arXiv:2308.06463. Retrieved from https:\/\/arxiv.org\/abs\/2308.06463"},{"key":"e_1_3_2_302_2","unstructured":"Zheng Yuan Hongyi Yuan Chuanqi Tan Wei Wang Songfang Huang and Fei Huang. 2023. RRHF: Rank responses to align language models with human feedback without tears. arXiv:2304.05302. Retrieved from https:\/\/arxiv.org\/abs\/2304.05302"},{"key":"e_1_3_2_303_2","unstructured":"Jiawei Liu Yifeng Ding Naman JainvHarm de Vries Leandro von Werra Arjun Guha Lingming Zhang Yuxiang Wei Federico Cassano. 2024. StarCoder2-Instruct: Fully Transparent and Permissive Self-Alignment for Code Generation. Retrieved from https:\/\/github.com\/bigcode-project\/starcoder2-self-align"},{"key":"e_1_3_2_304_2","unstructured":"Elad Ben Zaken Shauli Ravfogel and Yoav Goldberg. 2021. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv:2106.10199. Retrieved from https:\/\/arxiv.org\/abs\/2106.10199"},{"key":"e_1_3_2_305_2","unstructured":"Daoguang Zan Bei Chen Dejian Yang Zeqi Lin Minsu Kim Bei Guan Yongji Wang Weizhu Chen and Jian-Guang Lou. 2022. CERT: Continual pre-training on sketches for library-oriented code generation. arXiv:2206.06888. Retrieved from https:\/\/arxiv.org\/abs\/2206.06888"},{"key":"e_1_3_2_306_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.411"},{"key":"e_1_3_2_307_2","unstructured":"Daoguang Zan Ailun Yu Wei Liu Dong Chen Bo Shen Wei Li Yafen Yao Yongshun Gong Xiaolin Chen Bei Guan et al. 2024. CodeS: Natural language to code repository via multi-layer sketch. arXiv:2403.16443. Retrieved from https:\/\/arxiv.org\/abs\/2403.16443"},{"key":"e_1_3_2_308_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.151"},{"key":"e_1_3_2_309_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649850"},{"key":"e_1_3_2_310_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560425"},{"key":"e_1_3_2_311_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Zhang Qingru","year":"2023","unstructured":"Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_312_2","unstructured":"Shengyu Zhang Linfeng Dong Xiaoya Li Sen Zhang Xiaofei Sun Shuhe Wang Jiwei Li Runyi Hu Tianwei Zhang Fei Wu et al. 2023. Instruction tuning for large language models: A survey. arXiv:2308.10792. Retrieved from https:\/\/arxiv.org\/abs\/2308.10792"},{"key":"e_1_3_2_313_2","doi-asserted-by":"crossref","unstructured":"Shudan Zhang Hanlin Zhao Xiao Liu Qinkai Zheng Zehan Qi Xiaotao Gu Xiaohan Zhang Yuxiao Dong and Jie Tang. 2024. NaturalCodeBench: Examining coding performance mismatch on HumanEval and natural user prompts. arXiv:2405.04520. Retrieved from https:\/\/arxiv.org\/abs\/2405.04520","DOI":"10.18653\/v1\/2024.findings-acl.471"},{"key":"e_1_3_2_314_2","unstructured":"Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu Tingchen Fu Xinting Huang Enbo Zhao Yu Zhang Yulong Chen et al. 2023. Siren\u2019s song in the AI ocean: A survey on hallucination in large language models. arXiv:2309.01219. Retrieved from https:\/\/arxiv.org\/abs\/2309.01219"},{"key":"e_1_3_2_315_2","doi-asserted-by":"crossref","unstructured":"Yuntong Zhang Haifeng Ruan Zhiyu Fan and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous program improvement. arXiv:2404.05427. Retrieved from https:\/\/arxiv.org\/abs\/2404.05427","DOI":"10.1145\/3650212.3680384"},{"key":"e_1_3_2_316_2","unstructured":"Ziyin Zhang Chaoyu Chen Bingchang Liu Cong Liao Zi Gong Hang Yu Jianguo Li and Rui Wang. 2023. Unifying the perspectives of NLP and software engineering: A survey on language models for code. arXiv:2311.07989. Retrieved from https:\/\/arxiv.org\/abs\/2311.07989"},{"key":"e_1_3_2_317_2","unstructured":"Liang Zhao Xiaocheng Feng Xiachong Feng Bin Qin and Ting Liu. 2023. Length extrapolation of transformers: A survey from the perspective of position encoding. arXiv:2312.17044. Retrieved from https:\/\/arxiv.org\/abs\/2312.17044"},{"key":"e_1_3_2_318_2","unstructured":"Penghao Zhao Hailin Zhang Qinhan Yu Zhengren Wang Yunteng Geng Fangcheng Fu Ling Yang Wentao Zhang Jie Jiang and Bin Cui. 2024. Retrieval-augmented generation for AI-generated content: A survey. arXiv:2402.19473. Retrieved from https:\/\/arxiv.org\/abs\/2402.19473"},{"key":"e_1_3_2_319_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et al. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/arxiv.org\/abs\/2303.18223"},{"key":"e_1_3_2_320_2","article-title":"Judging LLM-as-a-judge with MT-bench and Chatbot Arena","volume":"36","author":"Zheng Lianmin","year":"2024","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_321_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_2_322_2","doi-asserted-by":"crossref","unstructured":"Tianyu Zheng Ge Zhang Tianhao Shen Xueling Liu Bill Yuchen Lin Jie Fu Wenhu Chen and Xiang Yue. 2024. OpenCodeInterpreter: Integrating code generation with execution and refinement. arXiv:2402.14658. Retrieved from https:\/\/arxiv.org\/abs\/2402.14658","DOI":"10.18653\/v1\/2024.findings-acl.762"},{"key":"e_1_3_2_323_2","first-page":"42403","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zheng Wenqing","year":"2023","unstructured":"Wenqing Zheng, S. P. Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023. Outline, then details: Syntactically guided coarse-to-fine code generation. In Proceedings of the International Conference on Machine Learning. PMLR, 42403\u201342419."},{"key":"e_1_3_2_324_2","unstructured":"Zibin Zheng Kaiwen Ning Yanlin Wang Jingwen Zhang Dewu Zheng Mingxi Ye and Jiachi Chen. 2023. A survey of large language models for code: Evolution benchmarking and future trends. arXiv:2311.10372. Retrieved from https:\/\/arxiv.org\/abs\/2311.10372"},{"key":"e_1_3_2_325_2","doi-asserted-by":"crossref","unstructured":"Li Zhong Zilong Wang and Jingbo Shang. 2024. LDB: A large language model debugger via verifying runtime execution step-by-step. arXiv:2402.16906. Retrieved from https:\/\/arxiv.org\/abs\/2402.16906","DOI":"10.18653\/v1\/2024.findings-acl.49"},{"key":"e_1_3_2_326_2","unstructured":"Aojun Zhou Ke Wang Zimu Lu Weikang Shi Sichun Luo Zipeng Qin Shaoqing Lu Anya Jia Linqi Song Mingjie Zhan et al. 2023. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. arXiv:2308.07921. Retrieved from https:\/\/arxiv.org\/abs\/2308.07921"},{"key":"e_1_3_2_327_2","unstructured":"Andy Zhou Kai Yan Michal Shlapentokh-Rothman Haohan Wang and Yu-Xiong Wang. 2023. Language agent tree search unifies reasoning acting and planning in language models. arXiv:2310.04406. Retrieved from https:\/\/arxiv.org\/abs\/2310.04406"},{"key":"e_1_3_2_328_2","article-title":"LIMA: Less is more for alignment","volume":"36","author":"Zhou Chunting","year":"2024","unstructured":"Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. LIMA: Less is more for alignment. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_329_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Zhou Denny","year":"2022","unstructured":"Denny Zhou, Nathanael Sch\u00e4rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_330_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Zhou Shuyan","year":"2022","unstructured":"Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig. 2022. DocPrompting: Generating code by retrieving the docs. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_331_2","unstructured":"Qihao Zhu Daya Guo Zhihong Shao Dejian Yang Peiyi Wang Runxin Xu Y. Wu Yukun Li Huazuo Gao Shirong Ma et al. 2024. DeepSeek-Coder-V2: Breaking the barrier of closed-source models in code intelligence. arXiv:2406.11931. Retrieved from https:\/\/arxiv.org\/abs\/2406.11931"},{"key":"e_1_3_2_332_2","first-page":"2232","volume-title":"Findings of the Association for Computational Linguistics: EACL 2024","author":"Zhuo Terry Yue","year":"2024","unstructured":"Terry Yue Zhuo. 2024. ICE-Score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, 2232\u20132242."},{"key":"e_1_3_2_333_2","unstructured":"Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu Ratnadira Widyasari Imam Nur Bani Yusuf Haolan Zhan Junda He Indraneil Paul et al. 2024. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. arXiv:2406.15877. Retrieved from https:\/\/arxiv.org\/abs\/2406.15877"},{"key":"e_1_3_2_334_2","unstructured":"Terry Yue Zhuo Armel Zebaze Nitchakarn Suppattarachai Leandro von Werra Harm de Vries Qian Liu and Niklas Muennighoff. 2024. Astraios: Parameter-efficient instruction tuning code large language models. arXiv:2401.00788. Retrieved from https:\/\/arxiv.org\/abs\/2401.00788"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3747588","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T16:33:12Z","timestamp":1769013192000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3747588"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,21]]},"references-count":333,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,2,28]]}},"alternative-id":["10.1145\/3747588"],"URL":"https:\/\/doi.org\/10.1145\/3747588","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,21]]},"assertion":[{"value":"2024-06-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}