{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,14]],"date-time":"2026-11-14T08:10:51Z","timestamp":1794643851646,"version":"4.9.1"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","license":[{"start":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T00:00:00Z","timestamp":1750550400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/publication-rights-and-licensing-policy"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>The capabilities of Large Language Models (LLMs) in code generation have been extensively studied, particularly for implementing target functionalities from natural-language descriptions. As an alternative to natural language, input-output (I\/O) examples provide an accessible, unambiguous, and flexible way to describe functionalities. However, their inherent diversity, opaqueness, and incompleteness impose greater challenges for understanding and implementing the target requirements. Therefore, generating code from I\/O examples (i.e., example-based code generation) provides a new perspective, allowing us to additionally evaluate LLMs\u2019 capability to infer target functionalities from limited information and to process new-form requirements. However, related research about LLMs in example-based code generation remains largely unexplored. To fill this gap, this paper presents the first comprehensive study on example-based code generation using LLMs. To address the incorrectness caused by the incompleteness of I\/O examples, we adopt an iterative evaluation framework and formalize the objective of example-based code generation as two sequential sub-objectives: generating code conforming to the given examples and generating code that successfully implements the target functionalities from (iteratively) given examples. We assess six state-of-the-art LLMs using a new benchmark of 172 diverse target functionalities (derived from HumanEval and CodeHunt). The results demonstrate that when requirements are described using iterative I\/O examples rather than natural language, the LLMs\u2019 score decreases by over 60%, indicating that example-based code generation remains challenging for the evaluated LLMs. Notably, the vast majority (even over 95%) of successfully implemented functionalities are achieved in the first round of the iterations, suggesting that the LLMs struggle to effectively utilize the iteratively supplemented requirements. Furthermore, we find that combining I\/O examples with even imprecise and fragmental natural language descriptions greatly improves LLM performance, and the selection of initial I\/O examples can also influence the score, suggesting opportunities for prompt optimization. These findings highlight the importance of early prompts during interactions and offer critical insights and implications for enhancing LLM-based code generation.<\/jats:p>","DOI":"10.1145\/3728947","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"1583-1606","source":"Crossref","is-referenced-by-count":5,"title":["The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-Based Code Generation"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2574-9774","authenticated-orcid":false,"given":"Yingjie","family":"Fu","sequence":"first","affiliation":[{"name":"School of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7519-5733","authenticated-orcid":false,"given":"Bozhou","family":"Li","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5403-3217","authenticated-orcid":false,"given":"Linyi","family":"Li","sequence":"additional","affiliation":[{"name":"Simon Fraser University, Burnaby, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7532-5550","authenticated-orcid":false,"given":"Wentao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6731-216X","authenticated-orcid":false,"given":"Tao","family":"Xie","sequence":"additional","affiliation":[{"name":"Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; School of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","unstructured":"Wasi Ahmad Md Golam Rahman Tushar Saikat Chakraborty and Kai-Wei Chang. 2023. AVATAR: A Parallel Corpus for Java-Python Program Translation. In Findings of the 2023 Association for Computational Linguistics. 2268\u20132281. https:\/\/doi.org\/10.48550\/arXiv.2108.11590 10.48550\/arXiv.2108.11590","DOI":"10.48550\/arXiv.2108.11590"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","unstructured":"Toufique Ahmed Kunal Suresh Pai Prem Devanbu and Earl T. Barr. 2024. Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). In Proceedings of the 2024 IEEE\/ACM International Conference on Software Engineering. 1\u201313. https:\/\/doi.org\/10.48550\/arXiv.2304.06815 10.48550\/arXiv.2304.06815","DOI":"10.48550\/arXiv.2304.06815"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program Synthesis with Large Language Models. https:\/\/doi.org\/10.48550\/arXiv.2108.07732 10.48550\/arXiv.2108.07732","DOI":"10.48550\/arXiv.2108.07732"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","unstructured":"M Balog AL Gaunt M Brockschmidt S Nowozin and D Tarlow. 2017. DeepCoder: Learning to Write Programs. In Proceedings of the 2017 International Conference on Learning Representations. https:\/\/doi.org\/10.48550\/arXiv.1611.01989 10.48550\/arXiv.1611.01989","DOI":"10.48550\/arXiv.1611.01989"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","unstructured":"Shaon Barman Sarah Chasins Rastislav Bodik and Sumit Gulwani. 2016. Ringer: Web Automation by Demonstration. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming Systems Languages and Applications. 748\u2013764. https:\/\/doi.org\/10.1145\/2983990.2984020 10.1145\/2983990.2984020","DOI":"10.1145\/2983990.2984020"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.172"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.14165"},{"key":"e_1_3_1_9_2","first-page":"209","article-title":"KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs","volume":"8","author":"Cadar Cristian","year":"2008","unstructured":"Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Proceedings of the 2008 USENIX Symposium on Operating Systems Design and Implementation, Vol. 8. 209\u2013224.","journal-title":"Proceedings of the 2008 USENIX Symposium on Operating Systems Design and Implementation"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","unstructured":"Pedro Calais and Lissa Franzini. 2023. Test-Driven Development Benefits Beyond Design Quality: Flow State and Developer Experience. In Proceedings of the 2023 IEEE\/ACM International Conference on Software Engineering: New Ideas and Emerging Results. 106\u2013111. https:\/\/doi.org\/10.1109\/ICSE-NIER58687.2023.00025 10.1109\/ICSE-NIER58687.2023.00025","DOI":"10.1109\/ICSE-NIER58687.2023.00025"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","unstructured":"Federico Cassano John Gouwar Daniel Nguyen Sydney Nguyen Luna Phipps-Costin Donald Pinckney Ming-Ho Yee Yangtian Zi Carolyn Jane Anderson Molly Q Feldman Arjun Guha Michael Greenberg and Abhinav Jangda. 2022. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. https:\/\/doi.org\/10.48550\/arXiv.2208.08227 10.48550\/arXiv.2208.08227","DOI":"10.48550\/arXiv.2208.08227"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.03109"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde De Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating Large Language Models Trained on Code. https:\/\/doi.org\/10.48550\/arXiv.2107.03374 10.48550\/arXiv.2107.03374","DOI":"10.48550\/arXiv.2107.03374"},{"key":"e_1_3_1_14_2","volume-title":"Watch What I Do: Programming by Demonstration","author":"Cypher Allen","year":"1993","unstructured":"Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky (Eds.). 1993. Watch What I Do: Programming by Demonstration. MIT Press."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","unstructured":"Robin David Luigi Coniglio Mariano Ceccato et al. 2020. QSynth-A Program Synthesis-Based Approach for Binary Code Deobfuscation. In Proceedings of the 2020 Workshop on Binary Analysis Research. https:\/\/doi.org\/10.14722\/bar.2020.23009 10.14722\/bar.2020.23009","DOI":"10.14722\/bar.2020.23009"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","unstructured":"DeepSeek AI Aixin Liu Bei Feng Bin Wang et al. 2024. DeepSeek-V2: A Strong Economical and Efficient Mixture-of-Experts Language Model. https:\/\/doi.org\/10.48550\/arXiv.2405.04434 10.48550\/arXiv.2405.04434","DOI":"10.48550\/arXiv.2405.04434"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","unstructured":"Xueying Du Mingwei Liu Kaixin Wang Hanlin Wang Junwei Liu Yixuan Chen Jiayi Feng Chaofeng Sha Xin Peng and Yiling Lou. 2023. ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. https:\/\/doi.org\/10.48550\/arXiv.2308.01861 10.48550\/arXiv.2308.01861","DOI":"10.48550\/arXiv.2308.01861"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian et al. 2024. The Llama 3 Herd of Models. https:\/\/doi.org\/10.48550\/arXiv.2407.21783 10.48550\/arXiv.2407.21783","DOI":"10.48550\/arXiv.2407.21783"},{"key":"e_1_3_1_19_2","unstructured":"Yingjie Fu Bozhou Li Linyi Li Wentao Zhang and Tao Xie. 2025. The InterCode Project. https:\/\/sites.google.com\/view\/intercodeproj"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2616877"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","unstructured":"Patrice Godefroid Nils Klarlund and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation. 213\u2013223. https:\/\/doi.org\/10.1145\/1064978.1065036 10.1145\/1064978.1065036","DOI":"10.1145\/1064978.1065036"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","unstructured":"Wenhan Xiong Grattafiori Alexandre D\u00e9fossez Jade Copet Faisal Azhar Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve et al. 2024. Code Llama: Open Foundation Models for Code. https:\/\/doi.org\/10.48550\/arXiv.2308.12950 10.48550\/arXiv.2308.12950","DOI":"10.48550\/arXiv.2308.12950"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","unstructured":"Sumit Gulwani. 2011. Automating String Processing in Spreadsheets Using Input-Output Examples. In Proceedings of the 2011 Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. 317\u2013330. https:\/\/doi.org\/10.1145\/1926385.1926423 10.1145\/1926385.1926423","DOI":"10.1145\/1926385.1926423"},{"key":"e_1_3_1_24_2","unstructured":"Sumit Gulwani. 2016. Programming by Examples (and its Applications in Data Wrangling). (2016). https:\/\/www.microsoft.com\/en-us\/research\/publication\/programming-examples-applications-data-wrangling\/"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2736282"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Yu Wu Y. K. Li Fuli Luo Yingfei Xiong Wenfeng Liang et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming \u2013 The Rise of Code Intelligence. https:\/\/doi.org\/10.48550\/arXiv.2401.14196 10.48550\/arXiv.2401.14196","DOI":"10.48550\/arXiv.2401.14196"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","unstructured":"Hossein Hajipour Mateusz Malinowski and Mario Fritz. 2021. IReEn: Reverse-Engineering of Black-Box Functions via Iterative Neural Program Synthesis. In Proceedings of the 2021 Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 143\u2013157. https:\/\/doi.org\/10.48550\/arXiv.2006.10720 10.48550\/arXiv.2006.10720","DOI":"10.48550\/arXiv.2006.10720"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","unstructured":"Yiyang Hao Ge Li Yongqiang Liu Xiaowei Miao He Zong Siyuan Jiang Yang Liu and He Wei. 2022. AixBench: A Code Generation Benchmark Dataset. https:\/\/doi.org\/10.48550\/arXiv.2206.13179 10.48550\/arXiv.2206.13179","DOI":"10.48550\/arXiv.2206.13179"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence with APPS. https:\/\/doi.org\/10.48550\/arXiv.2105.09938 10.48550\/arXiv.2105.09938","DOI":"10.48550\/arXiv.2105.09938"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2005.314"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","unstructured":"Juyong Jiang Fan Wang Jiasi Shen Sungju Kim and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. https:\/\/doi.org\/10.48550\/arXiv.2406.00515 10.48550\/arXiv.2406.00515","DOI":"10.48550\/arXiv.2406.00515"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","unstructured":"Nan Jiang Kevin Liu Thibaud Lutellier and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. In Proceedings of the 45th International Conference on Software Engineering. 1430\u20131442. https:\/\/doi.org\/10.48550\/arXiv.2302.05020 10.48550\/arXiv.2302.05020","DOI":"10.48550\/arXiv.2302.05020"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","unstructured":"Carlos E. Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In Proceedings of the 2024 International Conference on Learning Representations. https:\/\/doi.org\/10.48550\/arXiv.2310.06770 10.48550\/arXiv.2310.06770","DOI":"10.48550\/arXiv.2310.06770"},{"key":"e_1_3_1_34_2","unstructured":"Valentin Knappich. 2023. Tests4J benchmark: execution-based evaluation of context-aware language models for test case generation. Master\u2019s thesis."},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.11501"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","unstructured":"Vu Le and Sumit Gulwani. 2014. FlashExtract: A Framework for Data Extraction by Examples. In Proceedings of the 2014 ACM SIGPLAN Conference on Programming Language Design and Implementation. 542\u2013553. https:\/\/doi.org\/10.1145\/2594291.2594333 10.1145\/2594291.2594333","DOI":"10.1145\/2594291.2594333"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.03111"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","unstructured":"Rongao Li Jie Fu Bo-Wen Zhang Tao Huang Zhihong Sun Chen Lyu Guang Liu Zhi Jin and Ge Li. 2023. TACO: Topics in Algorithmic Code generation dataset. https:\/\/doi.org\/10.48550\/arXiv.2312.14852 10.48550\/arXiv.2312.14852","DOI":"10.48550\/arXiv.2312.14852"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","unstructured":"Wen-Ding Li and Kevin Ellis. 2024. Is Programming by Example Solved by LLMs? https:\/\/doi.org\/10.48550\/arXiv.2406.08316 10.48550\/arXiv.2406.08316","DOI":"10.48550\/arXiv.2406.08316"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_1_41_2","unstructured":"Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2024. EvalPlus Leaderboard. https:\/\/evalplus.github.io\/leaderboard.html"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.01210"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","unstructured":"Tianyang Liu Canwen Xu and Julian McAuley. 2023. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. https:\/\/doi.org\/10.48550\/arXiv.2306.03091 10.48550\/arXiv.2306.03091","DOI":"10.48550\/arXiv.2306.03091"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. https:\/\/doi.org\/10.48550\/arXiv.2402.19173 10.48550\/arXiv.2402.19173","DOI":"10.48550\/arXiv.2402.19173"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360569"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"Niklas Muennighoff Qian Liu Armel Zebaze Qinkai Zheng Binyuan Hui Terry Yue Zhuo Swayam Singh Xiangru Tang Leandro von Werra and Shayne Longpre. 2024. OctoPack: Instruction Tuning Code Large Language Models. https:\/\/doi.org\/10.48550\/arXiv.2308.07124 10.48550\/arXiv.2308.07124","DOI":"10.48550\/arXiv.2308.07124"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","unstructured":"Chandrakana Nandi Max Willsey Adam Anderson James R Wilcox Eva Darulova Dan Grossman and Zachary Tatlock. 2020. Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations. In Proceedings of the 2020 ACM SIGPLAN Conference on Programming Language Design and Implementation. 31\u201344. https:\/\/doi.org\/10.1145\/3385412.3386012 10.1145\/3385412.3386012","DOI":"10.1145\/3385412.3386012"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","unstructured":"Ansong Ni Srini Iyer Dragomir Radev Ves Stoyanov Wen-tau Yih Sida I. Wang and Xi Victoria Lin. 2023. LEVER: Learning to Verify Language-to-Code Generation with Execution. In Proceedings of the 2023 International Conference on Machine Learning. 26106\u201326128. https:\/\/doi.org\/10.48550\/arXiv.2302.08468 10.48550\/arXiv.2302.08468","DOI":"10.48550\/arXiv.2302.08468"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","unstructured":"Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. https:\/\/doi.org\/10.48550\/arXiv.2203.13474 10.48550\/arXiv.2203.13474","DOI":"10.48550\/arXiv.2203.13474"},{"key":"e_1_3_1_50_2","unstructured":"OpenAI. 2024. GPT-4o-mini. https:\/\/openai.com\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence\/ Large Language Model by OpenAI."},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","unstructured":"OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya et al. 2024. GPT-4 Technical Report. https:\/\/doi.org\/10.48550\/arXiv.2303.08774 10.48550\/arXiv.2303.08774","DOI":"10.48550\/arXiv.2303.08774"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","unstructured":"Rangeet Pan Vu Le Nachiappan Nagappan Sumit Gulwani Shuvendu Lahiri and Mike Kaufman. 2021. Can Program Synthesis be Used to Learn Merge Conflict Resolutions? An Empirical Analysis. In Proceedings of the 2021 International Conference on Software Engineering. 785\u2013796. https:\/\/doi.org\/10.48550\/arXiv.2103.02004 10.48550\/arXiv.2103.02004","DOI":"10.48550\/arXiv.2103.02004"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","unstructured":"Hafiz Arslan Ramzan Sadia Ramzan and Tehmina Kalsum. 2024. Test-Driven Development (TDD) in Small Software Development Teams: Advantages and Challenges. In Proceedings of the 2024 International Conference on Advancements in Computational Sciences. 1\u20135. https:\/\/doi.org\/10.1109\/ICACS60934.2024.10473291 10.1109\/ICACS60934.2024.10473291","DOI":"10.1109\/ICACS60934.2024.10473291"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","unstructured":"Mohammad Raza and Sumit Gulwani. 2017. Automated Data Extraction Using Predictive Program Synthesis. In Proceedings of the 2017 AAAI Conference on Artificial Intelligence. 882\u2013890. https:\/\/doi.org\/10.1609\/aaai.v31i1.10668 10.1609\/aaai.v31i1.10668","DOI":"10.1609\/aaai.v31i1.10668"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.4230\/OASIcs.PLATEAU.2018.3"},{"key":"e_1_3_1_56_2","unstructured":"John Schulman Barret Zoph Christina Kim Jacob Menick Jacob Hilton Jiayi Weng Juan Felipe Ceron Uribe Liam Fedus Luke Metz Michael Pokorny et al. 2022. ChatGPT: Optimizing Language Models for Dialogue. https:\/\/chatgpt.r4wand.eu.org\/"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/1095430.1081750"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","unstructured":"CodeGemma Team Heri Zhao Jeffrey Hui Joshua Howland Nam Nguyen Siqi Zuo Andrea Hu Christopher A Choquette-Choo Jingyue Shen Joe Kelley et al. 2024. CodeGemma: Open Code Models Based on Gemma. https:\/\/doi.org\/10.48550\/arXiv.2406.11409 10.48550\/arXiv.2406.11409","DOI":"10.48550\/arXiv.2406.11409"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","unstructured":"Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju Shreya Pathak Laurent Sifre Morgane Rivi\u00e8re Mihir Sanjay Kale Juliette Love et al. 2024. Gemma: Open Models Based on Gemini Research and Technology. https:\/\/doi.org\/10.48550\/arXiv.2403.08295 10.48550\/arXiv.2403.08295","DOI":"10.48550\/arXiv.2403.08295"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","unstructured":"Gemma Team Morgane Riviere Shreya Pathak Pier Giuseppe Sessa et al. 2024. Gemma 2: Improving Open Language Models at a Practical Size. https:\/\/doi.org\/10.48550\/arXiv.2408.00118 10.48550\/arXiv.2408.00118","DOI":"10.48550\/arXiv.2408.00118"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","unstructured":"Nikolai Tillmann Judith Bishop Nigel Horspool Daniel Perelman and Tao Xie. 2014. Code Hunt: Searching for Secret Code for Fun. In Proceedings of the 2014 International Workshop on Search-Based Software Testing. 23\u201326. https:\/\/doi.org\/10.1145\/2593833.2593838 10.1145\/2593833.2593838","DOI":"10.1145\/2593833.2593838"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","unstructured":"N. Tillmann and J. de Halleux. 2008. Pex-White Box Test Generation for .NET. In Proceedings of the 2008 International Conference on Tests and Proofs. 134\u2013153. https:\/\/doi.org\/10.1007\/978-3-540-79124-9_10 10.1007\/978-3-540-79124-9_10","DOI":"10.1007\/978-3-540-79124-9_10"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","unstructured":"Nikolai Tillmann Jonathan de Halleux and Tao Xie. 2014. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of the 2014 IEEE\/ACM International Conference on Automated Software Engineering. 385\u2013396. https:\/\/doi.org\/10.1145\/2642937.2642941 10.1145\/2642937.2642941","DOI":"10.1145\/2642937.2642941"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. https:\/\/doi.org\/10.48550\/arXiv.2307.09288 10.48550\/arXiv.2307.09288","DOI":"10.48550\/arXiv.2307.09288"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","unstructured":"Michele Tufano Shao Kun Deng Neel Sundaresan and Alexey Svyatkovskiy. 2022. Methods2Test: A Dataset of Focal Methods Mapped to Test Cases. In Proceedings of the 2022 International Conference on Mining Software Repositories. 299\u2013303. https:\/\/doi.org\/10.1145\/3524842.3528009 10.1145\/3524842.3528009","DOI":"10.1145\/3524842.3528009"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1911.09668"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","unstructured":"Jing Wei Sungdong Kim Hyunhoon Jung and Young-Ho Kim. 2024. Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data. In Proceedings of the 2024 ACM on Human-Computer Interaction. 1\u201335. https:\/\/doi.org\/10.1145\/3637364 10.1145\/3637364","DOI":"10.1145\/3637364"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","unstructured":"Yeming Wen Pengcheng Yin Kensen Shi Henryk Michalewski Swarat Chaudhuri and Alex Polozov. 2024. Grounding Data Science Code Generation with Input-Output Specifications. https:\/\/doi.org\/10.48550\/arXiv.2402.08073 10.48550\/arXiv.2402.08073","DOI":"10.48550\/arXiv.2402.08073"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","unstructured":"Tao Xie Nikolai Tillmann Jonathan de Halleux and Wolfram Schulte. 2009. Fitness-Guided Path Exploration in Dynamic Symbolic Execution. In Proceedings of the 2009 Annual IEEE\/IFIP International Conference on Dependable Systems and Networks. 359\u2013368. https:\/\/doi.org\/10.1109\/DSN.2009.5270315 10.1109\/DSN.2009.5270315","DOI":"10.1109\/DSN.2009.5270315"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591288"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","unstructured":"Hao Yu Bo Shen Dezhi Ran Jiaxin Zhang Qi Zhang Yuchi Ma Guangtai Liang Ying Li Qianxiang Wang and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the 2024 IEEE\/ACM International Conference on Software Engineering. 1\u201312. https:\/\/doi.org\/10.1145\/3597503.3623322 10.1145\/3597503.3623322","DOI":"10.1145\/3597503.3623322"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","unstructured":"Fengji Zhang Bei Chen Yue Zhang Jacky Keung Jin Liu Daoguang Zan Yi Mao Jian-Guang Lou and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2471\u20132484. https:\/\/doi.org\/10.48550\/arXiv.2303.12570 10.48550\/arXiv.2303.12570","DOI":"10.48550\/arXiv.2303.12570"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","unstructured":"Zhuosheng Zhang Aston Zhang Mu Li and Alex Smola. 2022. Automatic Chain of Thought Prompting in Large Language Models. https:\/\/doi.org\/10.48550\/arXiv.2210.03493 10.48550\/arXiv.2210.03493","DOI":"10.48550\/arXiv.2210.03493"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","unstructured":"Ming Zhu Aneesh Jain Karthik Suresh Roshan Ravindran Sindhu Tipirneni and Chandan K. Reddy. 2022. XLCoST: A Benchmark Dataset for Cross-Lingual Code Intelligence. https:\/\/doi.org\/10.48550\/arXiv.2206.08474 10.48550\/arXiv.2206.08474","DOI":"10.48550\/arXiv.2206.08474"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728947","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T13:48:44Z","timestamp":1787492924000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728947"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":73,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728947"],"URL":"https:\/\/doi.org\/10.1145\/3728947","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}