{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T21:44:12Z","timestamp":1774129452158,"version":"3.50.1"},"reference-count":141,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2025,3,22]],"date-time":"2025-03-22T00:00:00Z","timestamp":1742601600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:p>Machine learning (ML) pervades the field of Automated Program Repair (APR). Algorithms deploy neural machine translation and large language models (LLMs) to generate software patches, among other tasks. But, there are important differences between these applications of ML and earlier work, which complicates the task of ensuring that results are valid and likely to generalize. A challenge is that the most popular APR evaluation benchmarks were not designed with ML techniques in mind. This is especially true for LLMs, whose large and often poorly-disclosed training datasets may include problems on which they are evaluated.<\/jats:p>\n          <jats:p>This article reviews work in APR published in the field\u2019s top five venues since 2018, emphasizing emerging trends in the field, including the dramatic rise of ML models, including LLMs. ML-based articles are categorized along structural and functional dimensions, and a variety of issues are identified that these new methods raise. Importantly, data leakage and contamination concerns arise from the challenge of validating ML-based APR using existing benchmarks, which were designed before these techniques were popular. We discuss inconsistencies in evaluation design and performance reporting and offer pointers to solutions where they are available. Finally, we highlight promising new directions that the field is already taking.<\/jats:p>","DOI":"10.1145\/3704997","type":"journal-article","created":{"date-parts":[[2025,2,19]],"date-time":"2025-02-19T11:19:06Z","timestamp":1739963946000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Automated Program Repair: Emerging Trends Pose and Expose Problems for Benchmarks"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-2534-8700","authenticated-orcid":false,"given":"Joseph","family":"Renzullo","sequence":"first","affiliation":[{"name":"School of Computing and Augmented Intelligence, Arizona State University, Tempe, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1425-7811","authenticated-orcid":false,"given":"Pemma","family":"Reiter","sequence":"additional","affiliation":[{"name":"Arizona State University, Tempe, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6749-2204","authenticated-orcid":false,"given":"Westley","family":"Weimer","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5904-1646","authenticated-orcid":false,"given":"Stephanie","family":"Forrest","sequence":"additional","affiliation":[{"name":"Arizona State University, Tempe, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,3,22]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"NeurIPS. 2023. Machine Unlearning Challenge. Retrieved September 11 2023 from https:\/\/unlearning-challenge.github.io\/"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2944914"},{"key":"e_1_3_2_4_2","first-page":"2655","volume-title":"Proceedings of the NAACL-HLT 2021, Online, June 6-11, 2021.","author":"Ahmad Wasi Uddin","year":"2021","unstructured":"Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the NAACL-HLT 2021, Online, June 6-11, 2021.Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-T\u00fcr, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (Eds.), Association for Computational Linguistics, 2655\u20132668."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3212635"},{"issue":"5","key":"e_1_3_2_6_2","first-page":"99:1\u201399:30","article-title":"E-APR: Mapping the effectiveness of automated program repair techniques","volume":"26","author":"Aleti Aldeida","year":"2021","unstructured":"Aldeida Aleti and Matias Martinez. 2021. E-APR: Mapping the effectiveness of automated program repair techniques. Empirical Software Engineering 26, 5 (2021), 99:1\u201399:30.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_7_2","first-page":"143\u2013-153","volume-title":"Proceedings of the New Ideas, New Paradigms, and Reflections on Programming and Software","author":"Allamanis Miltiadis","year":"2019","unstructured":"Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the New Ideas, New Paradigms, and Reflections on Programming and Software (Athens, Greece). ACM, New York, NY, USA, 143\u2013-153."},{"key":"e_1_3_2_8_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program synthesis with large language Models. arXiv:2108.07732 [cs]"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.3390\/jimaging6060041"},{"key":"e_1_3_2_10_2","first-page":"907","volume-title":"Proceedings of the ASE 2020, Melbourne, Australia, September 21-25, 2020","author":"Benton Samuel","year":"2020","unstructured":"Samuel Benton, Xia Li, Yiling Lou, and Lingming Zhang. 2020. On the effectiveness of unified debugging: An extensive study on 16 program repair systems. In Proceedings of the ASE 2020, Melbourne, Australia, September 21-25, 2020. IEEE, 907\u2013918."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180219"},{"key":"e_1_3_2_12_2","first-page":"303","volume-title":"Proceedings of the Annual Computer Software and Applications Conference","author":"Bissyande Tegawende F.","year":"2013","unstructured":"Tegawende F. Bissyande, Ferdian Thung, David Lo, Lingxiao Jiang, and Laurent Reveillere. 2013. Popularity, interoperability, and impact of programming languages in 100,000 open source projects. In Proceedings of the Annual Computer Software and Applications Conference. IEEE, Kyoto, Japan, 303\u2013312."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317593"},{"key":"e_1_3_2_14_2","volume-title":"Advances in Neural Information Processing Systems","year":"2020","unstructured":"Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, and Tom Henighan. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33. NIPS."},{"key":"e_1_3_2_15_2","unstructured":"S\u00e9bastien Bubeck Varun Chandrasekaran Ronen Eldan Johannes Gehrke Eric Horvitz Ece Kamar Peter Lee Yin Tat Lee Yuanzhi Li Scott Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. arXiv:2303.12712 [cs]"},{"key":"e_1_3_2_16_2","first-page":"463","volume-title":"Proceedings of the Security and Privacy","author":"Cao Yinzhi","year":"2015","unstructured":"Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In Proceedings of the Security and Privacy. IEEE, San Jose, CA, 463\u2013480."},{"key":"e_1_3_2_17_2","first-page":"735","volume-title":"Proceedings of the Asia Conference on Computer and Communications Security","year":"2018","unstructured":"Yinzhi Cao, Alexander Fangxiao Yu, Andrew Aday, Eric Stahl, Jon Merwine, and Junfeng Yang. 2018. Efficient repair of polluted machine learning systems via causal unlearning. In Proceedings of the Asia Conference on Computer and Communications Security. ACM, Incheon Republic of Korea, 735\u2013747."},{"key":"e_1_3_2_18_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023","year":"2023","unstructured":"Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tram\u00e8r, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In Proceedings of the 11th International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net."},{"key":"e_1_3_2_19_2","volume-title":"Proceedings of the USENIX Security","author":"Carlini Nicholas","year":"2019","unstructured":"Nicholas Carlini, Chang Liu, \u00dalfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the USENIX Security. USENIX."},{"key":"e_1_3_2_20_2","first-page":"2633","volume-title":"Proceedings of the USENIX Security","year":"2021","unstructured":"Nicholas Carlini, Florian Tram\u00e8r, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, \u00dalfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In Proceedings of the USENIX Security. 2633\u20132650."},{"key":"e_1_3_2_21_2","first-page":"411","volume-title":"Proceedings of the Automated Software Engineering","author":"Cashin Padraic","year":"2019","unstructured":"Padraic Cashin, Carianne Martinez, Westley Weimer, and Stephanie Forrest. 2019. Understanding automatically-generated patches through symbolic invariant differences. In Proceedings of the Automated Software Engineering. IEEE, 411\u2013414."},{"key":"e_1_3_2_22_2","first-page":"3675","volume-title":"IEEE Trans. Software Eng.","year":"2023","unstructured":"Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation. IEEE Trans. Software Eng. 49, 7 (2023), 3675\u20133691."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3465055"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2970009"},{"key":"e_1_3_2_25_2","first-page":"831","volume-title":"IEEE Transactions on Software Engineering","year":"2023","unstructured":"Liushan Chen, Yu Pei, Minxue Pan, Tian Zhang, Qixin Wang, and Carlo A. Furia. 2023. Program repair with repeated learning. IEEE Transactions on Software Engineering 49, 2 (2023), 831\u2013848."},{"key":"e_1_3_2_26_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs]"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3147265"},{"key":"e_1_3_2_28_2","first-page":"1943","volume-title":"Transactions on Software Engineering","year":"2019","unstructured":"Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noel Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019. SEQUENCER: Sequence-to-sequence learning for end-to-end program repair. Transactions on Software Engineering 47, 9 (2019), 1943\u20131959."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3156637"},{"key":"e_1_3_2_30_2","unstructured":"Kyunghyun Cho Bart van Merrienboer Dzmitry Bahdanau and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In Syntax Semantics and Structure in Statistical Translation. Association for Computational Linguistics Doha 103\u2013111."},{"key":"e_1_3_2_31_2","first-page":"1","volume-title":"Journal of Machine Learning Research","year":"2023","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel, and Ruslan Salakhutdinov. 2023. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1\u2013113."},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition. IEEE Maimi 248\u2013255.","DOI":"10.1109\/CVPRW.2009.5206848"},{"key":"e_1_3_2_33_2","first-page":"4171","volume-title":"Proceedings of the NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1.","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1.Jill Burstein, Christy Doran, and Thamar Solorio (Eds.), Association for Computational Linguistics, 4171\u20134186."},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering.","author":"Ding Yangruibo","year":"2020","unstructured":"Yangruibo Ding, Baishakhi Ray, Premkumar Devanbu, and Vincent J. Hellendoorn. 2020. Patching as translation: The data and the metaphor. In Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering.ACM, Virtual Event."},{"key":"e_1_3_2_35_2","first-page":"1469","volume-title":"Proceedings of the International Conference on Software Engineering","author":"Fan Zhiyu","year":"2023","unstructured":"Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In Proceedings of the International Conference on Software Engineering. IEEE, 1469\u20131481."},{"key":"e_1_3_2_36_2","series-title":"Findings of ACL","first-page":"1536","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020","author":"Cohn Trevor","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020(Findings of ACL, Vol. EMNLP 2020), Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1536\u20131547."},{"key":"e_1_3_2_37_2","first-page":"1997","volume-title":"Proceedings of the SIGIR 2020","author":"Fr\u00f6be Maik","year":"2020","unstructured":"Maik Fr\u00f6be, Janek Bevendorff, Jan Heinrich Reimer, Martin Potthast, and Matthias Hagen. 2020. Sampling bias due to near-duplicates in learning to rank. In Proceedings of the SIGIR 2020. ACM, Virtual Event China, 1997\u20132000."},{"key":"e_1_3_2_38_2","first-page":"935","volume-title":"Proceedings of the ESEC\/FSE","author":"Fu Michael","year":"2022","unstructured":"Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. VulRepair: A T5-based automated software vulnerability repair. In Proceedings of the ESEC\/FSE. ACM, Singapore Singapore, 935\u2013947."},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/SPW.2018.00016","volume-title":"Proceedings of the 2018 IEEE Security and Privacy Workshops.","author":"Gao Ji","year":"2018","unstructured":"Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In Proceedings of the 2018 IEEE Security and Privacy Workshops.IEEE, 50\u201356."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2755013"},{"key":"e_1_3_2_41_2","first-page":"1262","volume-title":"Proceedings of the ASE 2019, San Diego, CA, USA, November 11-15, 2019","author":"Ghanbari Ali","year":"2019","unstructured":"Ali Ghanbari. 2019. Toward practical automatic program repair. In Proceedings of the ASE 2019, San Diego, CA, USA, November 11-15, 2019. IEEE, 1262\u20131264."},{"key":"e_1_3_2_42_2","first-page":"1118","volume-title":"Proceedings of the ASE 2019, San Diego, CA, USA, November 11-15, 2019","author":"Ghanbari Ali","year":"2019","unstructured":"Ali Ghanbari and Lingming Zhang. 2019. PraPR: Practical program repair via bytecode mutation. In Proceedings of the ASE 2019, San Diego, CA, USA, November 11-15, 2019. IEEE, 1118\u20131121."},{"issue":"4","key":"e_1_3_2_43_2","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1007\/s10664-021-10100-7","article-title":"A comprehensive study of code-removal patches in automated program repair","volume":"27","author":"Ginelli Davide","year":"2022","unstructured":"Davide Ginelli, Matias Martinez, Leonardo Mariani, and Martin Monperrus. 2022. A comprehensive study of code-removal patches in automated program repair. Empir. Softw. Eng. 27, 4 (2022), 97.","journal-title":"Empir. Softw. Eng."},{"key":"e_1_3_2_44_2","doi-asserted-by":"crossref","first-page":"1768","DOI":"10.1145\/3510003.3510620","volume-title":"Proceedings of the 44th International Conference on Software Engineering","author":"Gissurarson Matth\u00edas P\u00e1ll","year":"2022","unstructured":"Matth\u00edas P\u00e1ll Gissurarson, Leonhard Applis, Annibale Panichella, Arie Van Deursen, and David Sands. 2022. PropR: Property-based automatic program repair. In Proceedings of the 44th International Conference on Software Engineering. ACM, Pittsburgh Pennsylvania, 1768\u20131780."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387455"},{"key":"e_1_3_2_46_2","first-page":"888","volume-title":"Proceedings of the ESEC\/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018","author":"Hua Jinru","year":"2018","unstructured":"Jinru Hua, Mengshi Zhang, Kaiyuan Wang, and Sarfraz Khurshid. 2018. SketchFix: A tool for automated program repair approach using lazy candidate generation. In Proceedings of the ESEC\/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018. Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.), ACM, 888\u2013891."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180245"},{"key":"e_1_3_2_48_2","first-page":"3344","volume-title":"Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops.","year":"2020","unstructured":"Nikita Jaipuria, Xianling Zhang, Rohan Bhasin, Mayar Arafa, Punarjay Chakravarty, Shubham Shrivastava, Sagar Manglani, and Vidya N. Murali. 2020. Deflating dataset bias using synthetic data augmentation. In Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops.IEEE, Seattle, WA, USA, 3344\u20133353."},{"key":"e_1_3_2_49_2","first-page":"1430","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023","author":"Jiang Nan","year":"2023","unstructured":"Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1430\u20131442."},{"key":"e_1_3_2_50_2","first-page":"1251","volume-title":"Proceedings of the ICSE 2023, Melbourne, Australia, May 14-20, 2023","year":"2023","unstructured":"Nan Jiang, Thibaud Lutellier, Yiling Lou, Lin Tan, Dan Goldwasser, and Xiangyu Zhang. 2023. KNOD: Domain knowledge distilled tree decoder for automated program repair. In Proceedings of the ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1251\u20131263."},{"key":"e_1_3_2_51_2","first-page":"1161","volume-title":"Proceedings of the International Conference on Software Engineering","author":"Jiang Nan","year":"2021","unstructured":"Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. CURE: Code-aware neural machine translation for automatic program repair. In Proceedings of the International Conference on Software Engineering. IEEE, Madrid, Spain, 1161\u20131173."},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","unstructured":"Yuan Jiang Xiaohong Su Christoph Treude and Tiantian Wang. 2022. Hierarchical semantic-aware neural code representation. Journal of Systems and Software 191 (2022) 111355.","DOI":"10.1016\/j.jss.2022.111355"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2628055"},{"issue":"4","key":"e_1_3_2_54_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2382577.2382579","article-title":"Leakage in data mining: Formulation, detection, and avoidance","volume":"6","author":"Kaufman Shachar","year":"2012","unstructured":"Shachar Kaufman, Saharon Rosset, Claudia Perlich, and Ori Stitelman. 2012. Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data 6, 4 (2012), 1\u201321.","journal-title":"ACM Transactions on Knowledge Discovery from Data"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3067156"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09742-5"},{"key":"e_1_3_2_57_2","first-page":"1980","volume-title":"Empir. Softw. Eng.","year":"2020","unstructured":"Anil Koyuncu, Kui Liu, Tegawend\u00e9 F. Bissyand\u00e9, Dongsun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2020. FixMiner: Mining relevant fix patterns for automated program repair. Empir. Softw. Eng. 25, 3 (2020), 1980\u20132024."},{"key":"e_1_3_2_58_2","first-page":"314","volume-title":"Proceedings of the ESEC\/FSE","year":"2019","unstructured":"Anil Koyuncu, Kui Liu, Tegawend\u00e9 F. Bissyand\u00e9, Dongsun Kim, Martin Monperrus, Jacques Klein, and Yves Le Traon. 2019. iFixR: Bug report driven program repair. In Proceedings of the ESEC\/FSE. ACM, Tallinn Estonia, 314\u2013325."},{"key":"e_1_3_2_59_2","first-page":"1","volume-title":"Proceedings of the 2023 International Joint Conference on Neural Networks.","author":"Laroca Rayson","year":"2023","unstructured":"Rayson Laroca, Valter Estevam, Alceu S. Britto, Rodrigo Minetto, and David Menotti. 2023. Do we train on test data? The impact of near-duplicates on license plate recognition. In Proceedings of the 2023 International Joint Conference on Neural Networks.IEEE, Gold Coast, Australia, 1\u20138."},{"key":"e_1_3_2_60_2","doi-asserted-by":"crossref","unstructured":"Thanh Le-Cong Duc-Minh Luong Xuan Bach D. Le David Lo Nhat-Hoa Tran Bui Quang-Huy and Quyet-Thang Huynh. 2023. Invalidator: Automated patch correctness assessment via semantic and syntactic reasoning. IEEE Transactions on Software Engineering 49 6 (2023) 1\u201320.","DOI":"10.1109\/TSE.2023.3255177"},{"key":"e_1_3_2_61_2","first-page":"1236","volume-title":"Transactions on Software Engineering","year":"2015","unstructured":"Claire Le Goues, Neal Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. 2015. The manybugs and introclass benchmarks for automated repair of c programs. Transactions on Software Engineering 41, 12 (2015), 1236\u20131256."},{"issue":"1","key":"e_1_3_2_62_2","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/TSE.2011.104","article-title":"GenProg: A generic method for automatic software repair","volume":"38","author":"Goues Claire Le","year":"2012","unstructured":"Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A generic method for automatic software repair. Transactions on Software Engineering 38, 1 (2012), 54\u201372.","journal-title":"Transactions on Software Engineering"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318162"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387904.3389268"},{"key":"e_1_3_2_65_2","first-page":"8424","volume-title":"Proceedings of the ACL 2022, Dublin, Ireland, May 22-27, 2022.","author":"Muresan Smaranda","year":"2022","unstructured":"Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the ACL 2022, Dublin, Ireland, May 22-27, 2022.Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.), ACL, 8424\u20138445."},{"key":"e_1_3_2_66_2","first-page":"7871","volume-title":"Proceedings of the ACL 2020","year":"2020","unstructured":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the ACL 2020. ACL, Online, 7871\u20137880."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3153522"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3560422"},{"key":"e_1_3_2_69_2","first-page":"1092","volume-title":"Science","year":"2022","unstructured":"Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\u00e9mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien De Masson d.Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando De Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092\u20131097."},{"key":"e_1_3_2_70_2","first-page":"602","volume-title":"Proceedings of the International Conference on Software Engineering","author":"Li Yi","year":"2020","unstructured":"Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. DLFix: Context-based code transformation learning for automated program repair. In Proceedings of the International Conference on Software Engineering. ACM, Seoul, South Korea, 602\u2013614."},{"key":"e_1_3_2_71_2","first-page":"55","volume-title":"Proceedings of the Systems, Programming, Languages, and Applications: Software for Humanity","author":"Lin Derrick","year":"2017","unstructured":"Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings of the Systems, Programming, Languages, and Applications: Software for Humanity. ACM, 55\u201356."},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2884955"},{"key":"e_1_3_2_73_2","first-page":"31","volume-title":"Proceedings of the International Symposium on Software Testing and Analysis","author":"Liu Kui","year":"2019","unstructured":"Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawend\u00e9 F. Bissyand\u00e9. 2019. TBar: Revisiting template-based automated program repair. In Proceedings of the International Symposium on Software Testing and Analysis. ACM, Beijing, China, 31\u201342."},{"key":"e_1_3_2_74_2","first-page":"615","volume-title":"Proceedings of the ICSE","year":"2020","unstructured":"Kui Liu, Shangwen Wang, Anil Koyuncu, Kisub Kim, Tegawend\u00e9 F. Bissyand\u00e9, Dongsun Kim, Peng Wu, Jacques Klein, Xiaoguang Mao, and Yves Le Traon. 2020. On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs. In Proceedings of the ICSE. ACM, Seoul South Korea, 615\u2013627."},{"key":"e_1_3_2_75_2","first-page":"13","volume-title":"Prophet: Automatic Patch Generation via Learning from Successful Patches","author":"Long Fan","year":"2015","unstructured":"Fan Long and Martin Rinard. 2015. Prophet: Automatic Patch Generation via Learning from Successful Patches. Technical Report MIT-CSAIL-TR-2015-027. CSAIL, Massachusetts Institute of Technology. 13 pages."},{"key":"e_1_3_2_76_2","first-page":"1","volume-title":"Proceedings of the ACM on Programming Languages","year":"2017","unstructured":"Cristina V. Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. D\u00e9j\u00e0Vu: A map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 1\u201328."},{"key":"e_1_3_2_77_2","doi-asserted-by":"crossref","unstructured":"Minh-Thang Luong Hieu Pham and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. arXiv:1508.04025 [cs]","DOI":"10.18653\/v1\/D15-1166"},{"key":"e_1_3_2_78_2","first-page":"101","volume-title":"Proceedings of the International Symposium on Software Testing and Analysis","year":"2020","unstructured":"Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020. CoCoNuT: Combining context-aware neural translation models using ensemble for program repair. In Proceedings of the International Symposium on Software Testing and Analysis. ACM, Virtual Event, 101\u2013114."},{"key":"e_1_3_2_79_2","first-page":"1384","volume-title":"Proceedings of the EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022","author":"Madaan Aman","year":"2022","unstructured":"Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In Proceedings of the EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.), Association for Computational Linguistics, 1384\u20131403."},{"key":"e_1_3_2_80_2","first-page":"468","volume-title":"Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering.","author":"Madeiral Fernanda","year":"2019","unstructured":"Fernanda Madeiral, Simon Urli, Marcelo Maia, and Martin Monperrus. 2019. Bears: An extensible java bug benchmark for automatic program repair studies. In Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering.468\u2013478. arxiv:1901.06024 [cs]"},{"key":"e_1_3_2_81_2","first-page":"691","volume-title":"Proceedings of the International Conference on Software Engineering","author":"Mechtaev Sergey","year":"2016","unstructured":"Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the International Conference on Software Engineering. ACM, Austin, TX, USA, 691\u2013701."},{"issue":"6","key":"e_1_3_2_82_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi Ninareh","year":"2022","unstructured":"Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2022. A survey on bias and fairness in machine learning. Computing Surveys 54, 6 (2022), 1\u201335.","journal-title":"Computing Surveys"},{"key":"e_1_3_2_83_2","first-page":"1169","volume-title":"Proceedings of the ICSE","author":"Meng Xiangxin","year":"2022","unstructured":"Xiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2022. Improving fault localization and program repair with deep semantic features and transferred knowledge. In Proceedings of the ICSE. ACM, Pittsburgh Pennsylvania, 1169\u20131180."},{"key":"e_1_3_2_84_2","first-page":"1456","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering.","year":"2023","unstructured":"Xiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun, Xudong Liu, and Chunming Hu. 2023. Template-based neural program repair. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering.IEEE, Melbourne, Australia, 1456\u20131468."},{"key":"e_1_3_2_85_2","unstructured":"Martin Monperrus. 2018. The Living Review on Automated Program Repair. Technical Report. KTH Royal Institute of Technology."},{"key":"e_1_3_2_86_2","first-page":"1225","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering, Melbourne, Australia, May 14-20, 2023","author":"Motwani Manish","year":"2023","unstructured":"Manish Motwani and Yuriy Brun. 2023. Better automatic program repair by using bug reports and tests together. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering, Melbourne, Australia, May 14-20, 2023. IEEE, 1225\u20131237."},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9550-0"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2998785"},{"issue":"7","key":"e_1_3_2_89_2","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1007\/s10664-022-10223-5","article-title":"A controlled experiment of different code representations for learning-based program repair","volume":"27","author":"Namavar Marjane","year":"2022","unstructured":"Marjane Namavar, Noor Nashid, and Ali Mesbah. 2022. A controlled experiment of different code representations for learning-based program repair. Empirical Software Engineering 27, 7 (2022), 190.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_90_2","first-page":"1565","volume-title":"Proceedings of the ESEC\/FSE","author":"Nikitopoulos Georgios","year":"2021","unstructured":"Georgios Nikitopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. 2021. CrossVul: A cross-language vulnerability dataset with commit data. In Proceedings of the ESEC\/FSE. 1565\u20131569."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510040"},{"key":"e_1_3_2_92_2","first-page":"922","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2022, Singapore, Singapore, November 14-18, 2022.","author":"Oh Wonseok","year":"2022","unstructured":"Wonseok Oh and Hakjoo Oh. 2022. PyTER: Effective program repair for python type errors. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2022, Singapore, Singapore, November 14-18, 2022.Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (Eds.), ACM, 922\u2013934."},{"key":"e_1_3_2_93_2","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]"},{"key":"e_1_3_2_94_2","unstructured":"Nikhil Parasaram Earl T. Barr and Sergey Mechtaev. 2021. Trident: Controlling side effects in automated program repair. Transactions on Software Engineering 48 12 (2021) 4717\u20134732."},{"key":"e_1_3_2_95_2","first-page":"1264","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering.","author":"Parasaram Nikhil","year":"2023","unstructured":"Nikhil Parasaram, Earl T. Barr, and Sergey Mechtaev. 2023. Rete: Learning namespace representation for program repair. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering.IEEE, Melbourne, Australia, 1264\u20131276."},{"key":"e_1_3_2_96_2","first-page":"1291","volume-title":"Proceedings of the 16th ACM International Conference on Web Search and Data Mining","year":"2023","unstructured":"Charith Peris, Christophe Dupuy, Jimit Majmudar, Rahil Parikh, Sami Smaili, Richard Zemel, and Rahul Gupta. 2023. Privacy in the time of language models. In Proceedings of the 16th ACM International Conference on Web Search and Data Mining. 1291\u20131292."},{"key":"e_1_3_2_97_2","unstructured":"Jason Phang Herbie Bradley Leo Gao Louis Castricato and Stella Biderman. 2022. EleutherAI: Going beyond\u201d open science\u201d to\u201d science in the open\u201d. arXiv preprint arXiv:2210.06413 (2022)."},{"key":"e_1_3_2_98_2","unstructured":"Julian Aron Prenner and Romain Robbes. 2023. RunBugRun \u2013 An Executable Dataset for Automated Program Repair. arXiv:2304.01102 [cs.SE]"},{"key":"e_1_3_2_99_2","first-page":"5485","volume-title":"Journal of Machine Learning Research","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 1 (2020), 5485\u20135551."},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9564-7"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.01.008"},{"issue":"115","key":"e_1_3_2_102_2","first-page":"64","article-title":"A survey on software clone detection research","volume":"541","author":"Roy Chanchal Kumar","year":"2007","unstructured":"Chanchal Kumar Roy and James R. Cordy. 2007. A survey on software clone detection research. Queen\u2019s School of Computing TR 541, 115 (2007), 64\u201368.","journal-title":"Queen\u2019s School of Computing TR"},{"key":"e_1_3_2_103_2","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/3196398.3196473","volume-title":"Proceedings of the 15th International Conference on Mining Software Repositories - MSR \u201918","author":"Saha Ripon K.","year":"2018","unstructured":"Ripon K. Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R. Prasad. 2018. Bugs.Jar: A large-scale, diverse dataset of real-world java bugs. In Proceedings of the 15th International Conference on Mining Software Repositories - MSR \u201918. ACM Press, Gothenburg, Sweden, 10\u201313."},{"key":"e_1_3_2_104_2","first-page":"13","volume-title":"Proceedings of the International Conference on Software Engineering","author":"Saha Seemanta","year":"2019","unstructured":"Seemanta Saha, Ripon K. Saha, and Mukul R. Prasad. 2019. Harnessing evolution for multi-hunk program repair. In Proceedings of the International Conference on Software Engineering. IEEE, Montreal, QC, Canada, 13\u201324."},{"key":"e_1_3_2_105_2","unstructured":"Teven Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ili\u0107 Daniel Hesslow Roman Castagn\u00e9 Alexandra Sasha Luccioni Fran\u00e7ois Yvon Matthias Gall\u00e9 Jonathan Tow Alexander M. Rush Stella Biderman et\u00a0al. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv:2211.05100 [cs]"},{"key":"e_1_3_2_106_2","unstructured":"Rylan Schaeffer Brando Miranda and Sanmi Koyejo. 2023. Are Emergent Abilities of Large Language Models a Mirage? arXiv:2304.15004. Retrieved from https:\/\/arxiv.org\/abs\/2304.15004"},{"issue":"11","key":"e_1_3_2_107_2","doi-asserted-by":"crossref","first-page":"2673","DOI":"10.1109\/78.650093","article-title":"Bidirectional recurrent neural networks","volume":"45","author":"Schuster M.","unstructured":"M. Schuster and K.K. Paliwal. Nov.\/1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45, 11 (1997), 2673\u20132681.","journal-title":"IEEE Transactions on Signal Processing"},{"issue":"11","key":"e_1_3_2_108_2","doi-asserted-by":"crossref","first-page":"9549","DOI":"10.1609\/aaai.v35i11.17150","article-title":"Membership privacy for machine learning models through knowledge transfer","volume":"35","author":"Shejwalkar Virat","year":"2021","unstructured":"Virat Shejwalkar and Amir Houmansadr. 2021. Membership privacy for machine learning models through knowledge transfer. Proceedings of the AAAI Conference on Artificial Intelligence 35, 11 (2021), 9549\u20139557.","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"e_1_3_2_109_2","unstructured":"Ilya Sutskever Oriol Vinyals and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems 27 (2014)."},{"key":"e_1_3_2_110_2","doi-asserted-by":"crossref","unstructured":"Alexandre Tamborrino Nicola Pellicano Baptiste Pannier Pascal Voitot and Louise Naudin. 2020. Pre-Training Is (Almost) All You Need: An Application to Commonsense Reasoning.In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2020.acl-main.357"},{"key":"e_1_3_2_111_2","first-page":"180","volume-title":"Proceedings of the 39th International Conference on Software Engineering Companion","year":"2017","unstructured":"Shin Hwei Tan, Jooyong Yi, Sergey Mechtaev, and Abhik Roychoudhury. 2017. Codeflaws: A programming competition benchmark for evaluating automated program repair tools. In Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, 180\u2013182."},{"key":"e_1_3_2_112_2","first-page":"981","volume-title":"Proceedings of the Automated Software Engineering","year":"2020","unstructured":"Haoye Tian, Kui Liu, Abdoul Kader Kabor\u00e9, Anil Koyuncu, Li Li, Jacques Klein, and Tegawend\u00e9 F. Bissyand\u00e9. 2020. Evaluating representation learning of code changes for predicting patch correctness in program repair. In Proceedings of the Automated Software Engineering. ACM, Virtual Event Australia, 981\u2013992."},{"key":"e_1_3_2_113_2","first-page":"1","volume-title":"Proceedings of the Automated Software Engineering","year":"2022","unstructured":"Haoye Tian, Xunzhu Tang, Andrew Habib, Shangwen Wang, Kui Liu, Xin Xia, Jacques Klein, and Tegawend\u00e9 F. Bissyand\u00e9. 2022. Is this change the answer to that problem?: Correlating descriptions of bug and code changes for evaluating patch correctness. In Proceedings of the Automated Software Engineering. ACM, Rochester MI USA, 1\u201313."},{"key":"e_1_3_2_114_2","first-page":"1","volume-title":"ACM Transactions on Software Engineering and Methodology","year":"2019","unstructured":"Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology 28, 4 (2019), 1\u201329."},{"key":"e_1_3_2_115_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems Vol. 30. Curran Associates Inc. 5998\u20136008."},{"key":"e_1_3_2_116_2","first-page":"968","volume-title":"Proceedings of the Automated Software Engineering","year":"2020","unstructured":"Shangwen Wang, Ming Wen, Bo Lin, Hongjun Wu, Yihao Qin, Deqing Zou, Xiaoguang Mao, and Hai Jin. 2020. Automated patch correctness assessment: How far are we?. In Proceedings of the Automated Software Engineering. ACM, Virtual Event, 968\u2013980."},{"key":"e_1_3_2_117_2","first-page":"22964","volume-title":"Proceedings of the International Conference on Machine Learning","year":"2022","unstructured":"Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022. What language model architecture and pretraining objective work best for zero-shot generalization?. In Proceedings of the International Conference on Machine Learning. 22964\u201322984."},{"key":"e_1_3_2_118_2","unstructured":"Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter Fei Xia Ed H. Chi Quoc V. Le and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022) 24824\u201324837."},{"key":"e_1_3_2_119_2","unstructured":"Zhenmei Shi Junyi Wei Zhuoyan Xu and Yingyu Liang. 2024. Why larger language models do In-context learning differently? In International Conference on Machine Learning. PMLR 44991\u201345013."},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180233"},{"key":"e_1_3_2_121_2","first-page":"354","volume-title":"Proceedings of the ESEC\/FSE \u201921.","author":"Wong Chu-Pan","year":"2021","unstructured":"Chu-Pan Wong, Priscila Santiesteban, Christian K\u00e4stner, and Claire Le Goues. 2021. VarFix: Balancing edit expressiveness and search effectiveness in automated program repair. In Proceedings of the ESEC\/FSE \u201921.Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.), ACM, 354\u2013366."},{"issue":"8","key":"e_1_3_2_122_2","doi-asserted-by":"crossref","first-page":"8675","DOI":"10.1609\/aaai.v36i8.20846","article-title":"PUMA: Performance unchanged model augmentation for training data removal","volume":"36","author":"Wu Ga","year":"2022","unstructured":"Ga Wu, Masoud Hashemi, and Christopher Srinivasa. 2022. PUMA: Performance unchanged model augmentation for training data removal. Proceedings of the AAAI Conference on Artificial Intelligence 36, 8 (2022), 8675\u20138682.","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"e_1_3_2_123_2","first-page":"1482","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering","author":"Xia Chunqiu Steven","year":"2023","unstructured":"Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering. IEEE, Melbourne, Australia, 1482\u20131494."},{"key":"e_1_3_2_124_2","first-page":"959","volume-title":"Proceedings of the ESEC\/FSE","author":"Xia Chunqiu Steven","year":"2022","unstructured":"Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: Revisiting automated program repair via zero-shot learning. In Proceedings of the ESEC\/FSE. ACM, Singapore Singapore, 959\u2013971."},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_2_126_2","first-page":"309","volume-title":"IEEE Transactions on Software Engineering","year":"2022","unstructured":"Tongtong Xu, Liushan Chen, Yu Pei, Tian Zhang, Minxue Pan, and Carlo A. Furia. 2022. Restore: Retrospective fault localization enhancing automated program repair. IEEE Transactions on Software Engineering 48, 1 (2022), 309\u2013326."},{"key":"e_1_3_2_127_2","first-page":"512","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering","author":"Xu Xuezheng","year":"2019","unstructured":"Xuezheng Xu, Yulei Sui, Hua Yan, and Jingling Xue. 2019. VFix: Value-flow-guided precise program repair for null pointer dereferences. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering. IEEE, Montreal, QC, Canada, 512\u2013523."},{"key":"e_1_3_2_128_2","first-page":"1672","volume-title":"Proceedings of the ESEC\/FSE","author":"Yang Chen","year":"2021","unstructured":"Chen Yang. 2021. Accelerating redundancy-based program repair via code representation learning and adaptive patch filtering. In Proceedings of the ESEC\/FSE. ACM, Athens Greece, 1672\u20131674."},{"key":"e_1_3_2_129_2","first-page":"122","volume-title":"Empir. Softw. Eng.","year":"2021","unstructured":"Deheng Yang, Kui Liu, Dongsun Kim, Anil Koyuncu, Kisub Kim, Haoye Tian, Yan Lei, Xiaoguang Mao, Jacques Klein, and Tegawend\u00e9 F. Bissyand\u00e9. 2021. Where were the repair ingredients for Defects4j bugs? Empir. Softw. Eng. 26, 6 (2021), 122."},{"key":"e_1_3_2_130_2","first-page":"107:1\u2013107:13","volume-title":"Proceedings of the ASE 2022, Rochester, MI, USA, October 10-14, 2022","year":"2022","unstructured":"Deheng Yang, Xiaoguang Mao, Liqian Chen, Xuezheng Xu, Yan Lei, David Lo, and Jiayu He. 2022. TransplantFix: Graph differencing-based code transplantation for automated program repair. In Proceedings of the ASE 2022, Rochester, MI, USA, October 10-14, 2022. ACM, 107:1\u2013107:13."},{"key":"e_1_3_2_131_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3071750"},{"key":"e_1_3_2_132_2","first-page":"1","volume-title":"Proceedings of the Automated Software Engineering","author":"Ye He","year":"2022","unstructured":"He Ye, Matias Martinez, Xiapu Luo, Tao Zhang, and Martin Monperrus. 2022. SelfAPR: Self-supervised program repair with test execution diagnostics. In Proceedings of the Automated Software Engineering. ACM, Oakland Center, MI, US, 1\u201313."},{"key":"e_1_3_2_133_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09920-w"},{"key":"e_1_3_2_134_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510222"},{"key":"e_1_3_2_135_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9619-4"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2874648"},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.1007\/s42979-023-01767-4"},{"key":"e_1_3_2_138_2","first-page":"1","volume-title":"Proceedings of the Automated Software Engineering","author":"Zhang Jiyang","year":"2022","unstructured":"Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. CoditT5: Pretraining for source code and natural language editing. In Proceedings of the Automated Software Engineering. ACM, Rochester MI USA, 1\u201312."},{"key":"e_1_3_2_139_2","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong Yifan Du Chen Yang Yushuo Chen Zhipeng Chen Jinhao Jiang Ruiyang Ren Yifan Li Xinyu Tang Zikang Liu Peiyu Liu Jian-Yun Nie and Ji-Rong Wen. 2023. A Survey of Large Language Models. arXiv:2303.18223 [cs]"},{"key":"e_1_3_2_140_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556943"},{"key":"e_1_3_2_141_2","first-page":"341","volume-title":"Proceedings of the ESEC\/FSE","year":"2021","unstructured":"Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021. A syntax-guided edit decoder for neural program repair. In Proceedings of the ESEC\/FSE. ACM, Athens Greece, 341\u2013353."},{"key":"e_1_3_2_142_2","first-page":"1443","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering.","author":"Zhu Qihao","year":"2023","unstructured":"Qihao Zhu, Zeyu Sun, Wenjie Zhang, Yingfei Xiong, and Lu Zhang. 2023. Tare: Type-aware neural program repair. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering.IEEE, Melbourne, Australia, 1443\u20131455."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3704997","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3704997","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:01Z","timestamp":1750295881000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3704997"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,22]]},"references-count":141,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2025,8,31]]}},"alternative-id":["10.1145\/3704997"],"URL":"https:\/\/doi.org\/10.1145\/3704997","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,22]]},"assertion":[{"value":"2024-01-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}