{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T16:18:08Z","timestamp":1792253888898,"version":"4.4.0"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","license":[{"start":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T00:00:00Z","timestamp":1728345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2024,10,8]]},"abstract":"<jats:p>Code naturalness, which captures repetitiveness and predictability in programming languages, has proven valuable for various code-related tasks in software engineering. However, precisely measuring code naturalness remains a fundamental challenge. Existing methods measure code naturalness over individual lines of code while ignoring the deep semantic relations among different lines, e.g., program dependency, which may negatively affect the precision of the measure. Despite the intuitive appeal of extending the code naturalness measure to the code dependency domain (as there are some work that have initiated the utilization of code dependency for diverse code-related tasks), this assumption remains unexplored and warrants direct investigation. In this study, we aim to perform the first empirical study to investigate whether incorporating code dependency, instead of analyzing individual lines, can enhance the precision of measuring code naturalness.<\/jats:p>\n                  <jats:p>To achieve that, we first propose a new method named DAN for measuring code naturalness by incorporating the rich dependency information in the code. Specifically, DAN extracts multiple sequences of code lines by traversing the program dependency graph, where different code lines are connected by dependencies in each sequence, and then the code naturalness will be measured by taking each sequence as a whole. In this way, the dependency information can be well captured. Finally, we have conducted an extensive study to evaluate the influence of code dependency for measuring code naturalness with DAN, and compared it with the state-of-the-art methods under three emerging application scenarios of code naturalness. The results demonstrate that DAN can not only better distinguish natural and unnatural code, but also substantially boost two important downstream applications of code naturalness, i.e., distinguishing buggy and non-buggy code lines and data cleansing for training better code models, reflecting the significance of code dependency in measuring code naturalness.<\/jats:p>","DOI":"10.1145\/3689794","type":"journal-article","created":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T03:23:04Z","timestamp":1728357784000},"page":"2355-2377","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Dependency-Aware Code Naturalness"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0759-940X","authenticated-orcid":false,"given":"Chen","family":"Yang","sequence":"first","affiliation":[{"name":"Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3056-9962","authenticated-orcid":false,"given":"Junjie","family":"Chen","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1983-6572","authenticated-orcid":false,"given":"Jiajun","family":"Jiang","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3840-4988","authenticated-orcid":false,"given":"Yuliang","family":"Huang","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,10,8]]},"reference":[{"key":"e_1_3_1_2_1","unstructured":"Accessed: 2023. GrowingBugs. https:\/\/github.com\/jiangyanjie\/GrowingBugs\/."},{"key":"e_1_3_1_3_1","unstructured":"Accessed: 2023. javalang. https:\/\/github.com\/c2nes\/javalang\/."},{"key":"e_1_3_1_4_1","unstructured":"Accessed: 2023. OpenAI ChatGPT. https:\/\/chat.openai.com\/."},{"key":"e_1_3_1_5_1","unstructured":"Accessed: 2023. scikit-learn. https:\/\/scikit-learn.org\/stable\/."},{"key":"e_1_3_1_6_1","doi-asserted-by":"crossref","unstructured":"Miltiadis Allamanis Earl T Barr Christian Bird and Charles Sutton. 2014. Learning Natural Coding Conventions. ACM (2014).","DOI":"10.1145\/2635868.2635883"},{"key":"e_1_3_1_7_1","unstructured":"Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian Ming Tan Wasi Uddin Ahmad Shiqi Wang Qing Sun Mingyue Shang et al. 2022. Multi-lingual Evaluation of Code Generation Models. In The Eleventh International Conference on Learning Representations."},{"key":"e_1_3_1_8_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)."},{"key":"e_1_3_1_9_1","doi-asserted-by":"crossref","unstructured":"Teresa Busjahn Roman Bednarik Andrew Begel Martha Crosby James H Paterson Carsten Schulte Bonita Sharif and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE 255\u2013265.","DOI":"10.1109\/ICPC.2015.36"},{"issue":"12","key":"e_1_3_1_10_1","article-title":"Do programmers prefer predictable expressions in code","volume":"44","author":"Casalnuovo Casey","year":"2020","unstructured":"Casey Casalnuovo, Kevin Lee, Hulin Wang, Prem Devanbu, and Emily Morgan. 2020a. Do programmers prefer predictable expressions in code? Cognitive science 44, 12 (2020), e12921.","journal-title":"Cognitive science"},{"key":"e_1_3_1_11_1","volume-title":"Proceedings of the 42nd Annual Meeting of the Cognitive Science Society","author":"Casalnuovo Casey","year":"2020","unstructured":"Casey Casalnuovo, E Morgan, and P Devanbu. 2020b. Does surprisal predict code comprehension difficulty. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society. Cognitive Science Society Toronto, Canada."},{"key":"e_1_3_1_12_1","volume-title":"Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval","author":"Cavnar William B","year":"1994","unstructured":"William B Cavnar, John M Trenkle, et al. 1994. N-gram-based text categorization. In Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval, Vol. 161175. Las Vegas, NV, 14."},{"key":"e_1_3_1_13_1","doi-asserted-by":"crossref","unstructured":"Saikat Chakraborty Toufique Ahmed Yangruibo Ding Premkumar T Devanbu and Baishakhi Ray. 2022. NatGen: generative pre-training by \u201cnaturalizing\u201d source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 18\u201330.","DOI":"10.1145\/3540250.3549162"},{"key":"e_1_3_1_14_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021b. Evaluating large language models trained on code.(2021). arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_3_1_15_1","doi-asserted-by":"crossref","unstructured":"Songqiang Chen Shuo Jin and Xiaoyuan Xie. 2021a. Testing your question answering software via asking recursively. In 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE 104\u2013116.","DOI":"10.1109\/ASE51524.2021.9678670"},{"key":"e_1_3_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3569932"},{"key":"e_1_3_1_17_1","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1007\/978-3-030-27455-9_12","volume-title":"Search-Based Software Engineering: 11th International Symposium, SSBSE 2019, Tallinn, Estonia, August 31\u2013September 1, 2019, Proceedings 11.","author":"Dantas Altino","year":"2019","unstructured":"Altino Dantas, Eduardo F de Souza, Jerffeson Souza, and Celso G Camilo-Junior. 2019. Code naturalness to assist search space exploration in search-based program repair methods. In Search-Based Software Engineering: 11th International Symposium, SSBSE 2019, Tallinn, Estonia, August 31\u2013September 1, 2019, Proceedings 11. Springer, 164\u2013170."},{"key":"e_1_3_1_18_1","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang et al. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. 1536\u20131547.","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_19_1","unstructured":"Suriya Gunasekar Yi Zhang Jyoti Aneja Caio C\u00e9sar Teodoro Mendes Allie Del Giorno Sivakanth Gopi Mojan Javaheripi Piero Kauffmann Gustavo de Rosa Olli Saarikivi et al. 2023. Textbooks Are All You Need. arXiv preprint arXiv:2306.11644 (2023)."},{"key":"e_1_3_1_20_1","unstructured":"Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang LIU Shujie Long Zhou Nan Duan Alexey Svyatkovskiy Shengyu Fu et al. 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. In International Conference on Learning Representations."},{"key":"e_1_3_1_21_1","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y Wu YK Li et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming\u2013The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024)."},{"key":"e_1_3_1_22_1","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021)."},{"key":"e_1_3_1_23_1","unstructured":"Steffen Herbold Alexander Trautsch Benjamin Ledel Alireza Aghamohammadi Taher Ahmed Ghaleb Kuljit Kaur Chahal Tim Bossenmaier Bhaveet Nagaria Philip Makedonski Matin Nili Ahmadabadi Kristof Szabados Helge Spieker Matej Madeja Nathaniel Hoy Valentina Lenarduzzi Shangwen Wang Gema Rodr\u00edguez-P\u00e9rez Ricardo Colomo-Palacios Roberto Verdecchia Paramvir Singh Yihao Qin Debasish Chakroborti Willard Davis Vijay Walunj Hongjun Wu Diego Marcilio Omar Alam Abdullah Aldaeej Idan Amit Burak Turhan Simon Eismann Anna-Katharina Wickert Ivano Malavolta Matus Sulir Fatemeh Fard Austin Z. Henley Stratos Kourtzanidis Eray Tuzun Christoph Treude Simin Maleki Shamasbi Ivan Pashchenko Marvin Wyrich James Davis Alexander Serebrenik Ella Albrecht Ethem Utku Aktas Daniel Str\u00fcber and Johannes Erbel. 2020. Large-Scale Manual Validation of Bug Fixing Commits: A Fine-grained Analysis of Tangling. arXiv:2011.06244 [cs.SE]"},{"key":"e_1_3_1_24_1","doi-asserted-by":"crossref","unstructured":"Abram Hindle Earl T Barr Zhendong Su Mark Gabel and Premkumar Devanbu. 2012. On the naturalness of software. In 2012 34th International Conference on Software Engineering (ICSE). IEEE.","DOI":"10.1109\/ICSE.2012.6227135"},{"key":"e_1_3_1_25_1","doi-asserted-by":"crossref","unstructured":"Yanjie Jiang Hui Liu Yuxia Zhang Weixing Ji Hao Zhong and Lu Zhang. 2022. Do bugs lead to unnaturalness of source code?. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1085\u20131096.","DOI":"10.1145\/3540250.3549149"},{"key":"e_1_3_1_26_1","doi-asserted-by":"crossref","unstructured":"Ren\u00e9 Just Darioush Jalali and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. 437\u2013440.","DOI":"10.1145\/2610384.2628055"},{"key":"e_1_3_1_27_1","doi-asserted-by":"crossref","unstructured":"Ahmed Khanfir Matthieu Jimenez Mike Papadakis and Yves Le Traon. 2022. Codebert-nt: code naturalness via codebert. In IEEE 22nd International Conference on Software Quality Reliability and Security (QRS). IEEE 936\u2013947.","DOI":"10.1109\/QRS57517.2022.00098"},{"key":"e_1_3_1_28_1","doi-asserted-by":"publisher","DOI":"10.3390\/info10040150"},{"key":"e_1_3_1_29_1","first-page":"21314","article-title":"Coderl: Mastering code generation through pretrained models and deep reinforcement learning","volume":"35","author":"Le Hung","year":"2022","unstructured":"Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 21314\u201321328.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_30_1","doi-asserted-by":"crossref","unstructured":"Xia Li Wei Li Yuqun Zhang and Lingming Zhang. 2019. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. 169\u2013180.","DOI":"10.1145\/3293882.3330574"},{"key":"e_1_3_1_31_1","doi-asserted-by":"crossref","unstructured":"Edward Loper and Steven Bird. 2002. NLTK: the Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics-Volume 1. 63\u201370.","DOI":"10.3115\/1118108.1118117"},{"key":"e_1_3_1_32_1","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1145\/3524842.3528456","volume-title":"Proceedings of the 19th International Conference on Mining Software Repositories(MSR \u201922)","author":"Ma Wei","year":"2022","unstructured":"Wei Ma, Mengjie Zhao, Ezekiel Soremekun, Qiang Hu, Jie M. Zhang, Mike Papadakis, Maxime Cordy, Xiaofei Xie, and Yves Le Traon. 2022. GraphCode2Vec: Generic Code Embedding via Lexical and Program Dependence Analyses. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR \u201922). Association for Computing Machinery, New York, NY, USA, 524\u2013536. https:\/\/doi.org\/10.1145\/3524842.3528456 10.1145\/3524842.3528456"},{"key":"e_1_3_1_33_1","unstructured":"Erik Nijkamp Bo Pang Hiroaki Hayashi Lifu Tu Huan Wang Yingbo Zhou Silvio Savarese and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)."},{"key":"e_1_3_1_34_1","unstructured":"Theo X Olausson Jeevana Priya Inala Chenglong Wang Jianfeng Gao and Armando Solar-Lezama. 2023. Demystifying GPT Self-Repair for Code Generation. arXiv preprint arXiv:2306.09896 (2023)."},{"key":"e_1_3_1_35_1","doi-asserted-by":"crossref","unstructured":"Jibesh Patra and Michael Pradel. 2021. Semantic bug seeding: a learning-based approach for creating realistic bugs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 906\u2013918.","DOI":"10.1145\/3468264.3468623"},{"issue":"1","key":"e_1_3_1_36_1","first-page":"71","article-title":"The Language Instinct: The New Science of Language and Mind","volume":"5","author":"Pilkington Adrian","year":"1996","unstructured":"Adrian Pilkington. 1996. The Language Instinct: The New Science of Language and Mind: by Steven Pinker, 1994, The Penguin Press, London pp. 494, ISBN 0 713 99099 6 (hbk.). Language and Literature 5, 1 (1996), 71\u201374.","journal-title":"The Penguin Press, London pp. 494, ISBN 0 713 99099 6 (hbk.). Language and Literature"},{"issue":"8","key":"e_1_3_1_37_1","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.","journal-title":"OpenAI blog"},{"key":"e_1_3_1_38_1","doi-asserted-by":"crossref","unstructured":"Musfiqur Rahman Dharani Palani and Peter C Rigby. 2019. Natural software revisited. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE 37\u201348.","DOI":"10.1109\/ICSE.2019.00022"},{"key":"e_1_3_1_39_1","doi-asserted-by":"crossref","unstructured":"Baishakhi Ray Vincent Hellendoorn Saheel Godhane Zhaopeng Tu Alberto Bacchelli and Premkumar Devanbu. 2016. On the\" naturalness\" of buggy code. In Proceedings of the 38th International Conference on Software Engineering. 428\u2013439.","DOI":"10.1145\/2884781.2884848"},{"key":"e_1_3_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2775051.2677009"},{"key":"e_1_3_1_41_1","first-page":"6","article-title":"Code completion with statistical language models","volume":"49","author":"Raychev Veselin","year":"2014","unstructured":"Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. Acm Sigplan Notices 49, 6 (2014).","journal-title":"Acm Sigplan Notices"},{"key":"e_1_3_1_42_1","unstructured":"Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan Ming Zhou Ambrosio Blanco and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. CoRR abs\/2009.10297 (2020)."},{"key":"e_1_3_1_43_1","doi-asserted-by":"crossref","unstructured":"Devjeet Roy Sarah Fakhoury and Venera Arnaoudova. 2021. Reassessing automatic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105\u20131116.","DOI":"10.1145\/3468264.3468588"},{"key":"e_1_3_1_44_1","unstructured":"Baptiste Roziere Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_3_1_45_1","unstructured":"Chenyao Suo Junjie Chen Shuang Liu Jiajun Jiang Yingquan Zhao and Jianrong Wang. 2024. Fuzzing MLIR Compiler Infrastructure via Operation Dependency Analysis. In Proceedings of the 33nd ACM SIGSOFT International Symposium on Software Testing and Analysis."},{"key":"e_1_3_1_46_1","unstructured":"Zhao Tian and Junjie Chen. 2023. Test-Case-Driven Programming Understanding in Large Language Models for Better Code Generation. arXiv:2309.16120 [cs.SE]"},{"key":"e_1_3_1_47_1","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_1_48_1","doi-asserted-by":"crossref","unstructured":"Zhaopeng Tu Zhendong Su and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 269\u2013280.","DOI":"10.1145\/2635868.2635875"},{"key":"e_1_3_1_49_1","unstructured":"Raja Vall\u00e9e-Rai Phong Co Etienne Gagnon Laurie Hendren Patrick Lam and Vijay Sundaresan. 1999. Soot-a Java bytecode optimization framework. In Proceedings of the 1999 conference of the Centre for Advanced Studies on Collaborative research. 13."},{"key":"e_1_3_1_50_1","doi-asserted-by":"crossref","unstructured":"Yue Wang Weishi Wang Shafiq Joty and Steven CH Hoi. 2021a. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 8696\u20138708.","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_1_51_1","doi-asserted-by":"crossref","unstructured":"Zan Wang Hanmo You Junjie Chen Yingyi Zhang Xuyuan Dong and Wenbin Zhang. 2021b. Prioritizing test inputs for deep neural networks via mutation analysis. In 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). IEEE 397\u2013409.","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_1_52_1","doi-asserted-by":"crossref","unstructured":"Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials (2007) 1\u20133.","DOI":"10.1002\/9780471462422.eoct979"},{"issue":"2","key":"e_1_3_1_53_1","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1553-2712.2004.tb01433.x","article-title":"Advanced statistics: understanding medical record review (MRR) studies","volume":"11","author":"Worster Andrew","year":"2004","unstructured":"Andrew Worster and Ted Haines. 2004. Advanced statistics: understanding medical record review (MRR) studies. Academic emergency medicine 11, 2 (2004), 187\u2013192.","journal-title":"Academic emergency medicine"},{"key":"e_1_3_1_54_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i21.30361"},{"key":"e_1_3_1_55_1","unstructured":"Ming Yan Junjie Chen Jie M Zhang Xuejie Cao Chen Yang and Mark Harman. 2023. Coco: Testing code generation systems via concretized instructions. arXiv preprint arXiv:2308.13319 (2023)."},{"key":"e_1_3_1_56_1","doi-asserted-by":"crossref","unstructured":"Meng Yan Xin Xia Yuanrui Fan Ahmed E Hassan David Lo and Shanping Li. 2020. Just-in-time defect identification and localization: A two-phase framework. IEEE Transactions on Software Engineering 48 1 (2020) 82\u2013101.","DOI":"10.1109\/TSE.2020.2978819"},{"key":"e_1_3_1_57_1","doi-asserted-by":"crossref","unstructured":"Chen Yang Junjie Chen Xingyu Fan Jiajun Jiang and Jun Sun. 2023. Silent Compiler Bug De-duplication via Three-Dimensional Analysis. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 677\u2013689.","DOI":"10.1145\/3597926.3598087"},{"key":"e_1_3_1_58_1","unstructured":"Chen Yang Junjie Chen Jiajun Jiang and Yuliang Huang. 2024a. Dependency-aware code naturalness. https:\/\/doi.org\/10.5281\/zenodo.12783666 10.5281\/zenodo.12783666"},{"key":"e_1_3_1_59_1","unstructured":"Chen Yang Junjie Chen Bin Lin Jianyi Zhou and Ziqi Wang. 2024b. Enhancing LLM-based Test Generation for Hard-to- Cover Branches via Program Analysis. arXiv preprint arXiv:2404.04966 (2024)."},{"key":"e_1_3_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3630010"},{"key":"e_1_3_1_61_1","unstructured":"Daoguang Zan Bei Chen Dejian Yang Zeqi Lin Minsu Kim Bei Guan Yongji Wang Weizhu Chen and Jian-Guang Lou. 2022a. CERT: Continual Pre-training on Sketches for Library-oriented Code Generation. In The 2022 International Joint Conference on Artificial Intelligence."},{"key":"e_1_3_1_62_1","unstructured":"Daoguang Zan Bei Chen Fengji Zhang Dianjie Lu Bingchao Wu Bei Guan Yongji Wang and Jian-Guang Lou. 2022b. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.09420 (2022)."},{"key":"e_1_3_1_63_1","unstructured":"Shun Zhang Zhenfang Chen Yikang Shen Mingyu Ding Joshua B Tenenbaum and Chuang Gan. 2023. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510 (2023)."},{"key":"e_1_3_1_64_1","doi-asserted-by":"crossref","unstructured":"Qinkai Zheng Xiao Xia Xu Zou Yuxiao Dong Shan Wang Yufei Xue Zihan Wang Lei Shen Andi Wang Yang Li et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023).","DOI":"10.1145\/3580305.3599790"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3689794","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3689794","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T09:11:01Z","timestamp":1770196261000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3689794"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,8]]},"references-count":63,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2024,10,8]]}},"alternative-id":["10.1145\/3689794"],"URL":"https:\/\/doi.org\/10.1145\/3689794","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,8]]},"assertion":[{"value":"2024-04-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}