{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T19:01:16Z","timestamp":1776884476283,"version":"3.51.2"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"abstract":"<jats:p>Large Language Models (LLMs) have shown impressive In-Context Learning (ICL) ability in code generation. LLMs take a prompt context consisting of a few demonstration examples and a new requirement as input, and output new programs without any parameter update. Existing studies have found that the performance of ICL-based code generation heavily depends on the quality of demonstration examples and thus arises research on selecting demonstration examples: given a new requirement, a few demonstration examples are selected from a candidate pool, where LLMs are expected to learn the pattern hidden in these selected demonstration examples. Existing approaches are mostly based on heuristics or randomly selecting examples. However, the distribution of randomly selected examples usually varies greatly, making the performance of LLMs less robust. The heuristics retrieve examples by only considering textual similarities of requirements, leading to sub-optimal performance.<\/jats:p>\n          <jats:p>\n            To fill this gap, we propose a\n            <jats:bold>L<\/jats:bold>\n            arge language model-\n            <jats:bold>A<\/jats:bold>\n            ware selection approach for\n            <jats:bold>I<\/jats:bold>\n            n-context-\n            <jats:bold>L<\/jats:bold>\n            earning-based code generation named LAIL. LAIL uses LLMs themselves to select examples. It requires LLMs themselves to label a candidate example as a positive example or a negative example for a requirement. Positive examples are helpful for LLMs to generate correct programs, while negative examples are trivial and should be ignored. Based on the labeled positive and negative data, LAIL trains a model-aware retriever to learn the preference of LLMs and select demonstration examples that LLMs need. During the inference, given a new requirement, LAIL uses the trained retriever to select a few examples and feed them into LLMs to generate desired programs. We apply LAIL to four widely used LLMs and evaluate it on five code generation datasets. Extensive experiments demonstrate that LAIL outperforms the state-of-the-art (SOTA) baselines by 11.58%, 3.33%, and 5.07% on CodeGen-Multi-16B, 1.32%, 2.29%, and 1.20% on CodeLlama-34B, and achieves 4.38%, 2.85%, and 2.74% improvements on Text-davinci-003 in terms of Pass@1 at MBJP, MBPP, and MBCPP, respectively. In addition to function-level code generation, LAIL improves the performance of LLMs on DevEval, a repository-level code generation dataset, which achieves 10.04%, 8.12%, and 4.63% improvements compared to the SOTA baselines at Pass@1, 3, and 5 on CodeLlama-7B. Human evaluation further verifies that the generated programs of LAIL are superior in correctness, code quality, and maintainability. Besides, LAIL has satisfactory transferability across different LLMs and datasets, where the retriever learned on one LLM (dataset) can be transferred to other LLMs (datasets).\n          <\/jats:p>","DOI":"10.1145\/3715908","type":"journal-article","created":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T15:31:29Z","timestamp":1740756689000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Large Language Model-Aware In-Context Learning for Code Generation"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9411-971X","authenticated-orcid":false,"given":"Jia","family":"Li","sequence":"first","affiliation":[{"name":"Key Lab of High Confidence Software Technology (Peking University), MoE, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4162-2119","authenticated-orcid":false,"given":"Chongyang","family":"Tao","sequence":"additional","affiliation":[{"name":"Beihang University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5579-8852","authenticated-orcid":false,"given":"Jia","family":"Li","sequence":"additional","affiliation":[{"name":"Key Lab of High Confidence Software Technology (Peking University), MoE, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5828-0186","authenticated-orcid":false,"given":"Ge","family":"Li","sequence":"additional","affiliation":[{"name":"Key Lab of High Confidence Software Technology (Peking University), MoE, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1087-226X","authenticated-orcid":false,"given":"Zhi","family":"Jin","sequence":"additional","affiliation":[{"name":"Key Lab of High Confidence Software Technology (Peking University), MoE, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0324-4591","authenticated-orcid":false,"given":"Huangzhao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Key Lab of High Confidence Software Technology (Peking University), MoE, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7000-6909","authenticated-orcid":false,"given":"Zheng","family":"Fang","sequence":"additional","affiliation":[{"name":"Key Lab of High Confidence Software Technology (Peking University), MoE, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3905-8133","authenticated-orcid":false,"given":"Fang","family":"Liu","sequence":"additional","affiliation":[{"name":"The State Key Laboratory of Software Development Environment (SKLSDE), SEI, School of Computer Science &amp; Engineering, Beihang University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3212695"},{"key":"e_1_2_1_3_1","volume-title":"et\u00a0al","author":"Athiwaratkun Ben","year":"2022","unstructured":"Ben Athiwaratkun, Sanjay\u00a0Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi\u00a0Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et\u00a0al. 2022. Multi-lingual Evaluation of Code Generation Models. arXiv preprint arXiv:2210.14868 (2022)."},{"key":"e_1_2_1_4_1","volume-title":"et\u00a0al","author":"Austin Jacob","year":"2021","unstructured":"Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et\u00a0al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)."},{"key":"e_1_2_1_5_1","volume-title":"Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code. arXiv preprint arXiv:2206.01335","author":"Barei\u00df Patrick","year":"2022","unstructured":"Patrick Barei\u00df, Beatriz Souza, Marcelo d\u2019Amorim, and Michael Pradel. 2022. Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code. arXiv preprint arXiv:2206.01335 (2022)."},{"key":"e_1_2_1_6_1","volume-title":"Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata 58","author":"Black Sid","year":"2021","unstructured":"Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata 58 (2021)."},{"key":"e_1_2_1_7_1","volume-title":"Generative code modeling with graphs. arXiv preprint arXiv:1805.08490","author":"Brockschmidt Marc","year":"2018","unstructured":"Marc Brockschmidt, Miltiadis Allamanis, Alexander\u00a0L Gaunt, and Oleksandr Polozov. 2018. Generative code modeling with graphs. arXiv preprint arXiv:1805.08490 (2018)."},{"key":"e_1_2_1_8_1","volume-title":"et\u00a0al","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared\u00a0D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877\u20131901."},{"key":"e_1_2_1_9_1","volume-title":"et\u00a0al","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared\u00a0D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877\u20131901."},{"key":"e_1_2_1_10_1","unstructured":"ChatGPT. 2022. https:\/\/platform.openai.com\/docs\/models\/gpt-3-5."},{"key":"e_1_2_1_11_1","volume-title":"Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397","author":"Chen Bei","year":"2022","unstructured":"Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397 (2022)."},{"key":"e_1_2_1_12_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et\u00a0al.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De\u00a0Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et\u00a0al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_2_1_13_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et\u00a0al.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de\u00a0Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et\u00a0al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_2_1_14_1","volume-title":"International conference on machine learning. PMLR, 1597\u20131607","author":"Chen Ting","year":"2020","unstructured":"Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597\u20131607."},{"key":"e_1_2_1_15_1","volume-title":"Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128","author":"Chen Xinyun","year":"2023","unstructured":"Xinyun Chen, Maxwell Lin, Nathanael Sch\u00e4rli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023)."},{"key":"e_1_2_1_16_1","unstructured":"CodeParrot. 2022. https:\/\/huggingface.co\/codeparrot\/codeparrot."},{"key":"e_1_2_1_17_1","volume-title":"Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829","author":"Coleman Cody","year":"2019","unstructured":"Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. 2019. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829 (2019)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111734"},{"key":"e_1_2_1_19_1","first-page":"7665","article-title":"A multi-view multi-task learning framework for multi-variate time series forecasting","volume":"35","author":"Deng Jinliang","year":"2022","unstructured":"Jinliang Deng, Xiusi Chen, Renhe Jiang, Xuan Song, and Ivor\u00a0W Tsang. 2022. A multi-view multi-task learning framework for multi-variate time series forecasting. IEEE Transactions on Knowledge and Data Engineering 35, 8 (2022), 7665\u20137680.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_2_1_20_1","volume-title":"Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting","author":"Deng Jinliang","year":"2024","unstructured":"Jinliang Deng, Xiusi Chen, Renhe Jiang, Du Yin, Yi Yang, Xuan Song, and Ivor\u00a0W Tsang. 2024. Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting. IEEE Transactions on Knowledge and Data Engineering (2024)."},{"key":"e_1_2_1_21_1","volume-title":"Coarse-to-fine decoding for neural semantic parsing. arXiv preprint arXiv:1805.04793","author":"Dong Li","year":"2018","unstructured":"Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. arXiv preprint arXiv:1805.04793 (2018)."},{"key":"e_1_2_1_22_1","volume-title":"A survey for in-context learning. arXiv preprint arXiv:2301.00234","author":"Dong Qingxiu","year":"2022","unstructured":"Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022)."},{"key":"e_1_2_1_23_1","volume-title":"et\u00a0al","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et\u00a0al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)."},{"key":"e_1_2_1_24_1","volume-title":"et\u00a0al","author":"Gao Leo","year":"2020","unstructured":"Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et\u00a0al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)."},{"key":"e_1_2_1_25_1","volume-title":"Constructing Effective In-Context Demonstration for Code Intelligence Tasks: An Empirical Study. arXiv preprint arXiv:2304.07575","author":"Gao Shuzheng","year":"2023","unstructured":"Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, and Michael\u00a0R Lyu. 2023. Constructing Effective In-Context Demonstration for Code Intelligence Tasks: An Empirical Study. arXiv preprint arXiv:2304.07575 (2023)."},{"key":"e_1_2_1_26_1","unstructured":"GPT-3.5. 2022. https:\/\/platform.openai.com\/docs\/deprecations."},{"key":"e_1_2_1_27_1","unstructured":"GPT-4. 2023. https:\/\/platform.openai.com\/docs\/models\/gpt-4-and-gpt-4-turbo. (2023)."},{"key":"e_1_2_1_28_1","unstructured":"GPT-CC. 2022. https:\/\/github.com\/CodedotAl\/gpt-code-clippy."},{"key":"e_1_2_1_29_1","unstructured":"GPT-J. 2023. https:\/\/github.com\/kingoflolz\/mesh-transformer-jax."},{"key":"e_1_2_1_30_1","volume-title":"et\u00a0al","author":"Guo Daya","year":"2020","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et\u00a0al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020)."},{"key":"e_1_2_1_31_1","volume-title":"The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751","author":"Holtzman Ari","year":"2019","unstructured":"Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)."},{"key":"e_1_2_1_32_1","volume-title":"Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019)."},{"key":"e_1_2_1_33_1","volume-title":"Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588","author":"Iyer Srinivasan","year":"2018","unstructured":"Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588 (2018)."},{"key":"e_1_2_1_34_1","volume-title":"Exploring dynamic selection of branch expansion orders for code generation. arXiv preprint arXiv:2106.00261","author":"Jiang Hui","year":"2021","unstructured":"Hui Jiang, Chulun Zhou, Fandong Meng, Biao Zhang, Jie Zhou, Degen Huang, Qingqiang Wu, and Jinsong Su. 2021. Exploring dynamic selection of branch expansion orders for code generation. arXiv preprint arXiv:2106.00261 (2021)."},{"key":"e_1_2_1_35_1","volume-title":"Self-planning code generation with large language model. arXiv preprint arXiv:2303.06689","author":"Jiang Xue","year":"2023","unstructured":"Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code generation with large language model. arXiv preprint arXiv:2303.06689 (2023)."},{"key":"e_1_2_1_36_1","volume-title":"International conference on machine learning. PMLR, 5110\u20135121","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In International conference on machine learning. PMLR, 5110\u20135121."},{"key":"e_1_2_1_37_1","volume-title":"Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800","author":"Levy Itay","year":"2022","unstructured":"Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800 (2022)."},{"key":"e_1_2_1_38_1","volume-title":"et\u00a0al","author":"Li Jia","year":"2024","unstructured":"Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, et\u00a0al. 2024. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv preprint arXiv:2405.19856 (2024)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00210"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2303.17780"},{"key":"e_1_2_1_41_1","volume-title":"et\u00a0al","author":"Li Raymond","year":"2023","unstructured":"Raymond Li, Loubna\u00a0Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et\u00a0al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)."},{"key":"e_1_2_1_42_1","volume-title":"et\u00a0al","author":"Li Yujia","year":"2022","unstructured":"Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\u00e9mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal\u00a0Lago, et\u00a0al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092\u20131097."},{"key":"e_1_2_1_43_1","volume-title":"Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493","author":"Li Yujia","year":"2015","unstructured":"Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 (2015)."},{"key":"e_1_2_1_44_1","volume-title":"Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744","author":"Ling Wang","year":"2016","unstructured":"Wang Ling, Edward Grefenstette, Karl\u00a0Moritz Hermann, Tom\u00e1\u0161 Ko\u010disk\u1ef3, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744 (2016)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00072"},{"key":"e_1_2_1_46_1","volume-title":"What Makes Good In-Context Examples for GPT-3? arXiv preprint arXiv:2101.06804","author":"Liu Jiachang","year":"2021","unstructured":"Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What Makes Good In-Context Examples for GPT-3? arXiv preprint arXiv:2101.06804 (2021)."},{"key":"e_1_2_1_47_1","unstructured":"Llama. 2023. https:\/\/huggingface.co\/meta-llama\/Llama-2-70b-chat. (2023)."},{"key":"e_1_2_1_48_1","volume-title":"et\u00a0al","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et\u00a0al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)."},{"key":"e_1_2_1_49_1","volume-title":"ICL: Demonstration-Retrieved In-context Learning. arXiv preprint arXiv:2305.14128","author":"Luo Man","year":"2023","unstructured":"Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent\u00a0Y Zhao. 2023. Dr. ICL: Demonstration-Retrieved In-context Learning. arXiv preprint arXiv:2305.14128 (2023)."},{"key":"e_1_2_1_50_1","volume-title":"WizardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv preprint arXiv:2306.08568","author":"Luo Ziyang","year":"2023","unstructured":"Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. arXiv preprint arXiv:2306.08568 (2023)."},{"key":"e_1_2_1_51_1","volume-title":"International Conference on Machine Learning. PMLR, 649\u2013657","author":"Maddison Chris","year":"2014","unstructured":"Chris Maddison and Daniel Tarlow. 2014. Structured generative models of natural source code. In International Conference on Machine Learning. PMLR, 649\u2013657."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/FLOSS.2007.10"},{"key":"e_1_2_1_53_1","volume-title":"On end-to-end program generation from user intention by deep neural networks. arXiv preprint arXiv:1510.07211","author":"Mou Lili","year":"2015","unstructured":"Lili Mou, Rui Men, Ge Li, Lu Zhang, and Zhi Jin. 2015. On end-to-end program generation from user intention by deep neural networks. arXiv preprint arXiv:1510.07211 (2015)."},{"key":"e_1_2_1_54_1","volume-title":"Neural sketch learning for conditional program generation. arXiv preprint arXiv:1703.05698","author":"Murali Vijayaraghavan","year":"2017","unstructured":"Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. 2017. Neural sketch learning for conditional program generation. arXiv preprint arXiv:1703.05698 (2017)."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311\u2013318","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_2_1_56_1","volume-title":"et\u00a0al","author":"Radford Alec","year":"2018","unstructured":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et\u00a0al. 2018. Improving language understanding by generative pre-training. (2018)."},{"key":"e_1_2_1_57_1","volume-title":"et\u00a0al","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9."},{"key":"e_1_2_1_58_1","volume-title":"et\u00a0al","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9."},{"key":"e_1_2_1_59_1","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter\u00a0J Liu, et\u00a0al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 140 (2020), 1\u201367.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_60_1","volume-title":"Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084","author":"Reimers Nils","year":"2019","unstructured":"Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)."},{"key":"e_1_2_1_61_1","volume-title":"Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297","author":"Ren Shuo","year":"2020","unstructured":"Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020)."},{"key":"e_1_2_1_62_1","volume-title":"et\u00a0al","author":"Robertson Stephen","year":"2009","unstructured":"Stephen Robertson, Hugo Zaragoza, et\u00a0al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends\u00ae in Information Retrieval 3, 4 (2009), 333\u2013389."},{"key":"e_1_2_1_63_1","volume-title":"et\u00a0al","author":"Rozi\u00e8re Baptiste","year":"2023","unstructured":"Baptiste Rozi\u00e8re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing\u00a0Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, et\u00a0al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_2_1_64_1","volume-title":"SmBoP: Semi-autoregressive bottom-up semantic parsing. arXiv preprint arXiv:2010.12412","author":"Rubin Ohad","year":"2020","unstructured":"Ohad Rubin and Jonathan Berant. 2020. SmBoP: Semi-autoregressive bottom-up semantic parsing. arXiv preprint arXiv:2010.12412 (2020)."},{"key":"e_1_2_1_65_1","volume-title":"The graph neural network model","author":"Scarselli Franco","year":"2008","unstructured":"Franco Scarselli, Marco Gori, Ah\u00a0Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE transactions on neural networks 20, 1 (2008), 61\u201380."},{"key":"e_1_2_1_66_1","volume-title":"Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909","author":"Sennrich Rico","year":"2015","unstructured":"Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 (2015)."},{"key":"e_1_2_1_67_1","volume-title":"et\u00a0al","author":"Su Hongjin","year":"2022","unstructured":"Hongjin Su, Jungo Kasai, Chen\u00a0Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah\u00a0A Smith, et\u00a0al. 2022. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 (2022)."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695000"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6430"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2404.17513"},{"key":"e_1_2_1_72_1","volume-title":"et\u00a0al","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et\u00a0al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_2_1_73_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan\u00a0N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_1_74_1","volume-title":"TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810.02720","author":"Yin Pengcheng","year":"2018","unstructured":"Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810.02720 (2018)."},{"key":"e_1_2_1_75_1","volume-title":"CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation. arXiv preprint arXiv:2206.06888","author":"Zan Daoguang","year":"2022","unstructured":"Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation. arXiv preprint arXiv:2206.06888 (2022)."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1"},{"key":"e_1_2_1_77_1","volume-title":"CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information. arXiv preprint arXiv:2202.07612","author":"Zhong Maosheng","year":"2022","unstructured":"Maosheng Zhong, Gen Liu, Hongwei Li, Jiangling Kuang, Jinshan Zeng, and Mingwen Wang. 2022. CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information. arXiv preprint arXiv:2202.07612 (2022)."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715908","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T15:32:57Z","timestamp":1740756777000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715908"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,28]]},"references-count":77,"alternative-id":["10.1145\/3715908"],"URL":"https:\/\/doi.org\/10.1145\/3715908","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,28]]},"assertion":[{"value":"2024-01-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"3715908"}}