{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T02:39:17Z","timestamp":1774579157383,"version":"3.50.1"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"abstract":"<jats:p>Unit testing validates the correctness of the units of the software system under test and serves as the cornerstone in improving software quality and reliability. To reduce manual efforts in writing unit tests, some techniques have been proposed to generate test assertions automatically, including deep learning (DL)-based, retrieval-based, and integration-based ones. Among them, recent integration-based approaches inherit from both DL-based and retrieval-based approaches and are considered state-of-the-art. Despite being promising, such integration-based approaches suffer from inherent limitations, such as retrieving assertions with lexical matching while ignoring meaningful code semantics, and generating assertions with a limited training corpus.<\/jats:p>\n          <jats:p>\n            In this paper, we propose a novel\n            <jats:bold>Retri<\/jats:bold>\n            eval-Augmented Deep Assertion\n            <jats:bold>Gen<\/jats:bold>\n            eration approach, namely RetriGen, based on a hybrid assertion retriever and a pre-trained language model (PLM)-based assertion generator. Given a focal-test, RetriGen first builds a hybrid assertion retriever to search for the most relevant test-assert pair from external codebases. The retrieval process takes both lexical similarity and semantical similarity into account via a token-based and an embedding-based retriever, respectively. RetriGen then treats assertion generation as a sequence-to-sequence task and designs a PLM-based assertion generator to predict a correct assertion with historical test-assert pairs and the retrieved external assertion. Although our concept is general and can be adapted to various off-the-shelf encoder-decoder PLMs, we implement RetriGen to facilitate assertion generation based on the recent CodeT5 model. We conduct extensive experiments to evaluate RetriGen against six state-of-the-art approaches across two large-scale datasets and two metrics. The experimental results demonstrate that RetriGen achieves 57.66% and 73.24% in terms of accuracy and CodeBLEU, outperforming all baselines with an average improvement of 50.66% and 14.14%, respectively. Furthermore, RetriGen generates 1598 and 1818 unique correct assertions that all baselines fail to produce, 3.71X and 4.58X more than the most recent approach\n            <jats:sc>EditAS<\/jats:sc>\n            . We also demonstrate that adopting other PLMs can provide substantial advancement,\n            <jats:italic>e.g.,<\/jats:italic>\n            four additionally-utilized PLMs outperform\n            <jats:sc>EditAS<\/jats:sc>\n            by 7.91%\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\sim\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            12.70% accuracy improvement, indicating the generalizability of RetriGen. Overall, our study highlights the promising future of fine-tuning off-the-shelf PLMs to generate accurate assertions by incorporating external knowledge sources.\n          <\/jats:p>","DOI":"10.1145\/3721128","type":"journal-article","created":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T15:31:29Z","timestamp":1740756689000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Improving Deep Assertion Generation via Fine-Tuning Retrieval-Augmented Pre-trained Language Models"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2495-3805","authenticated-orcid":false,"given":"Quanjun","family":"Zhang","sequence":"first","affiliation":[{"name":"Department of Computing Technologies, Swinburne University of Technology, Australia and State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9930-7111","authenticated-orcid":false,"given":"Chunrong","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2040-8496","authenticated-orcid":false,"given":"Yi","family":"Zheng","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6311-5184","authenticated-orcid":false,"given":"Yaxin","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1980-6277","authenticated-orcid":false,"given":"Yuan","family":"Zhao","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1769-6126","authenticated-orcid":false,"given":"Rubing","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Macau University of Science and Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4867-5416","authenticated-orcid":false,"given":"Jianyi","family":"Zhou","sequence":"additional","affiliation":[{"name":"Huawei Cloud Computing Technologies Co., Ltd., China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7868-5471","authenticated-orcid":false,"given":"Yun","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Computing Technologies, Swinburne University of Technology, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3736-4604","authenticated-orcid":false,"given":"Tao","family":"Zheng","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China and Shenzhen Research Institute of Nanjing University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9592-7022","authenticated-orcid":false,"given":"Zhenyu","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, China and Shenzhen Research Institute of Nanjing University, China"}]}],"member":"320","published-online":{"date-parts":[[2025,2,28]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2024. Hugging Face. site: https:\/\/huggingface.co\/."},{"key":"e_1_2_1_2_1","unstructured":"2024. OpenDDAL. site: https:\/\/github.com\/neradb\/openddal."},{"key":"e_1_2_1_3_1","unstructured":"2024. OWLAPI. site: https:\/\/github.com\/owlcs\/owlapi."},{"key":"e_1_2_1_4_1","unstructured":"2024. PyTorch. site: https:\/\/pytorch.org\/."},{"key":"e_1_2_1_5_1","volume-title":"2017 IEEE\/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track. 263\u2013272","author":"Almasi M\u00a0Moein","year":"2017","unstructured":"M\u00a0Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application. In 2017 IEEE\/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track. 263\u2013272."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 306\u2013317","author":"Barr T","year":"2014","unstructured":"Earl\u00a0T Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, and Federica Sarro. 2014. The Plastic Surgery Hypothesis. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 306\u2013317."},{"key":"e_1_2_1_8_1","volume-title":"ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572\u2013576","author":"Chen Yinghao","year":"2024","unstructured":"Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572\u2013576."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2014.11"},{"key":"e_1_2_1_10_1","volume-title":"2017 IEEE\/ACM 25th International Conference on Program Comprehension. 241\u2013250","author":"Dao Tung","year":"2017","unstructured":"Tung Dao, Lingming Zhang, and Na Meng. 2017. How Does Execution Information Help with Information-Retrieval Based Bug Localization?. In 2017 IEEE\/ACM 25th International Conference on Program Comprehension. 241\u2013250."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 44th International Conference on Software Engineering. 2130\u20132141","author":"Dinella Elizabeth","year":"2022","unstructured":"Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu\u00a0K Lahiri. 2022. TOGA: A Neural Method for Test Oracle Generation. In Proceedings of the 44th International Conference on Software Engineering. 2130\u20132141."},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1109\/32.988497","article-title":"Test Case Prioritization: A Family of Empirical Studies","volume":"28","author":"Elbaum Sebastian","year":"2002","unstructured":"Sebastian Elbaum, Alexey\u00a0G Malishevsky, and Gregg Rothermel. 2002. Test Case Prioritization: A Family of Empirical Studies. IEEE Transactions on Software Engineering 28, 2 (2002), 159\u2013182.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_13_1","volume-title":"Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE\/ACM International Conference on Software Engineering: Future of Software Engineering. IEEE, 31\u201353","author":"Fan Angela","year":"2023","unstructured":"Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie\u00a0M Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE\/ACM International Conference on Software Engineering: Future of Software Engineering. IEEE, 31\u201353."},{"key":"e_1_2_1_14_1","volume-title":"et\u00a0al","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et\u00a0al. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. 1536\u20131547."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. ACM, 416\u2013419","author":"Fraser Gordon","year":"2011","unstructured":"Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. ACM, 416\u2013419."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 935\u2013947","author":"Fu Michael","year":"2022","unstructured":"Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. VulRepair: A T5-based Automated Software Vulnerability Repair. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 935\u2013947."},{"key":"e_1_2_1_17_1","volume-title":"TestART: Improving LLM-based Unit Test via Co-evolution of Automated Generation and Repair Iteration. arXiv preprint arXiv:2408.03095","author":"Gu Siqi","year":"2024","unstructured":"Siqi Gu, Chunrong Fang, Quanjun Zhang, Fangyuan Tian, and Zhenyu Chen. 2024. TestART: Improving LLM-based Unit Test via Co-evolution of Automated Generation and Repair Iteration. arXiv preprint arXiv:2408.03095 (2024)."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 7212\u20137225","author":"Guo Daya","year":"2022","unstructured":"Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 7212\u20137225."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 9th International Conference on Learning Representations. 1\u201318","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao\u00a0Kun Deng, Colin\u00a0B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. In Proceedings of the 9th International Conference on Learning Representations. 1\u201318."},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1145\/566171.566207","article-title":"Is ISSTA Research Relevant to Industry","volume":"27","author":"Hartman Alan","year":"2002","unstructured":"Alan Hartman. 2002. Is ISSTA Research Relevant to Industry? ACM SIGSOFT Software Engineering Notes 27, 4 (2002), 205\u2013206.","journal-title":"ACM SIGSOFT Software Engineering Notes"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"He Yibo","year":"2024","unstructured":"Yibo He, Jiaming Huang, Hao Yu, and Tao Xie. 2024. An Empirical Study on Focal Methods in Deep-Learning-Based Approaches for Assertion Generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1750\u20131771."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616265"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 23rd International Symposium on Software Testing and Analysis. 437\u2013440","author":"Just Ren\u00e9","year":"2014","unstructured":"Ren\u00e9 Just, Darioush Jalali, and Michael\u00a0D Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proceedings of the 23rd International Symposium on Software Testing and Analysis. 437\u2013440."},{"key":"e_1_2_1_24_1","volume-title":"Dense Passage Retrieval for Open-Domain Question Answering. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP","author":"Karpukhin Vladimir","year":"2020","unstructured":"Vladimir Karpukhin, Barlas O\u011fuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen\u00a0Tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020. Association for Computational Linguistics, 6769\u20136781."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 45th International Conference on Software Engineering. 919\u2013931","author":"Lemieux Caroline","year":"2023","unstructured":"Caroline Lemieux, Jeevana\u00a0Priya Inala, Shuvendu\u00a0K Lahiri, and Siddhartha Sen. 2023. CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-Trained Large Language Models. In Proceedings of the 45th International Conference on Software Engineering. 919\u2013931."},{"key":"e_1_2_1_26_1","first-page":"9459","article-title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick S.\u00a0H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (2020), 9459\u20139474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_27_1","volume-title":"EDITSUM: A Retrieve-and-Edit Framework for Source Code Summarization. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering. 155\u2013166","author":"Li Jia","year":"2021","unstructured":"Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. 2021. EDITSUM: A Retrieve-and-Edit Framework for Source Code Summarization. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering. 155\u2013166."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1035\u20131047","author":"Li Zhiyu","year":"2022","unstructured":"Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et\u00a0al. 2022. Automating Code Review Activities by Large-Scale Pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1035\u20131047."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 589\u2013600","author":"Liu Zhongxin","year":"2023","unstructured":"Zhongxin Liu, Kui Liu, Xin Xia, and Xiaohu Yang. 2023. Towards More Realistic Evaluation for Neural Test Oracle Generation. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 589\u2013600."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409760"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin\u00a0B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao\u00a0Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks."},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","first-page":"1580","DOI":"10.1109\/TSE.2022.3183297","article-title":"Using Transfer Learning for Code-Related Tasks","volume":"49","author":"Mastropaolo Antonio","year":"2022","unstructured":"Antonio Mastropaolo, Nathan Cooper, David\u00a0Nader Palacio, Simone Scalabrino, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2022. Using Transfer Learning for Code-Related Tasks. IEEE Transactions on Software Engineering 49, 4 (2022), 1580\u20131598.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_33_1","volume-title":"2021 IEEE\/ACM 43rd International Conference on Software Engineering. IEEE, 336\u2013347","author":"Mastropaolo Antonio","year":"2021","unstructured":"Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David\u00a0Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. In 2021 IEEE\/ACM 43rd International Conference on Software Engineering. IEEE, 336\u2013347."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 45th International Conference on Software Engineering. IEEE, 2450\u20132462","author":"Nashid Noor","year":"2023","unstructured":"Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning. In Proceedings of the 45th International Conference on Software Engineering. IEEE, 2450\u20132462."},{"key":"e_1_2_1_35_1","volume-title":"Randoop: Feedback-Directed Random Testing for Java. In Companion to the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, 815\u2013816.","author":"Pacheco Carlos","year":"2007","unstructured":"Carlos Pacheco and Michael\u00a0D Ernst. 2007. Randoop: Feedback-Directed Random Testing for Java. In Companion to the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, 815\u2013816."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3469582"},{"key":"e_1_2_1_37_1","first-page":"2719","article-title":"Retrieval Augmented Code Generation and Summarization","volume":"2021","author":"Parvez Md\u00a0Rizwan","year":"2021","unstructured":"Md\u00a0Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval Augmented Code Generation and Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021. 2719\u20132734.","journal-title":"Findings of the Association for Computational Linguistics: EMNLP"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397383"},{"key":"e_1_2_1_39_1","volume-title":"The Economic Impacts of Inadequate Infrastructure for Software Testing","author":"Planning Strategic","year":"2002","unstructured":"Strategic Planning. 2002. The Economic Impacts of Inadequate Infrastructure for Software Testing. National Institute of Standards and Technology 1 (2002)."},{"key":"e_1_2_1_40_1","first-page":"5485","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter\u00a0J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485\u20135551.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_2_1_41_1","volume-title":"Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950","author":"Rozi\u00e8re Baptiste","year":"2023","unstructured":"Baptiste Rozi\u00e8re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing\u00a0Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D\u00e9fossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"Ryan Gabriel","year":"2024","unstructured":"Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali\u00a0Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM. Proceedings of the ACM on Software Engineering 1, FSE (2024), 951\u2013971."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3334955"},{"key":"e_1_2_1_44_1","volume-title":"GitBug-Java: A Reproducible Benchmark of Recent Java Bugs. In 2024 IEEE\/ACM 21st International Conference on Mining Software Repositories. 118\u2013122","author":"Silva Andr\u00e9","year":"2024","unstructured":"Andr\u00e9 Silva, Nuno Saavedra, and Martin Monperrus. 2024. GitBug-Java: A Reproducible Benchmark of Recent Java Bugs. In 2024 IEEE\/ACM 21st International Conference on Mining Software Repositories. 118\u2013122."},{"key":"e_1_2_1_45_1","volume-title":"Revisiting and Improving Retrieval-Augmented Deep Assertion Generation. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering. 1123\u20131135","author":"Sun Weifeng","year":"2023","unstructured":"Weifeng Sun, Hongyan Li, Meng Yan, Yan Lei, and Hongyu Zhang. 2023. Revisiting and Improving Retrieval-Augmented Deep Assertion Generation. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering. 1123\u20131135."},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","first-page":"1340","DOI":"10.1109\/TSE.2024.3382365","article-title":"ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation","volume":"50","author":"Tang Yutian","year":"2024","unstructured":"Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2024. ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation. IEEE Transactions on Software Engineering 50, 6 (2024), 1340\u20131359.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Runchu Tian Yining Ye Yujia Qin Xin Cong Yankai Lin Zhiyuan Liu and Maosong Sun. 2024. DebugBench: Evaluating Debugging Capability of Large Language Models. In Findings of the Association for Computational Linguistics. 4173\u20134198.","DOI":"10.18653\/v1\/2024.findings-acl.247"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3524481.3527220"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 44th International Conference on Software Engineering. 2291\u20132302","author":"Tufano Rosalia","year":"2022","unstructured":"Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022. Using Pre-Trained Models To Boost Code Review Automation. In Proceedings of the 44th International Conference on Software Engineering. 2291\u20132302."},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1109\/TSE.2024.3368208","article-title":"Software Testing With Large Language Models: Survey, Landscape, and Vision","volume":"50","author":"Wang Junjie","year":"2024","unstructured":"Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 04 (2024), 911\u2013936.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 146\u2013158","author":"Wang Weishi","year":"2023","unstructured":"Weishi Wang, Yue Wang, Shafiq Joty, and Steven\u00a0CH Hoi. 2023. RAP-Gen: Retrieval-Augmented Patch Generation with Codet5 for Automatic Program Repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 146\u2013158."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 8696\u20138708","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven\u00a0CH Hoi. 2021. CodeT5: Identifier-Aware Unified Pre-Trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 8696\u20138708."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1258\u20131268","author":"Wang Zejun","year":"2024","unstructured":"Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1258\u20131268."},{"key":"e_1_2_1_54_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3485275","article-title":"A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research","volume":"31","author":"Watson Cody","year":"2022","unstructured":"Cody Watson, Nathan Cooper, David\u00a0Nader Palacio, Kevin Moran, and Denys Poshyvanyk. 2022. A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research. ACM Transactions on Software Engineering and Methodology 31, 2 (2022), 1\u201358.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 1398\u20131409","author":"Watson Cody","year":"2020","unstructured":"Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On Learning Meaningful Assert Statements for Unit Test Cases. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 1398\u20131409."},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the 45th International Conference on Software Engineering. 1482\u20131494","author":"Xia Chunqiu\u00a0Steven","year":"2023","unstructured":"Chunqiu\u00a0Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Program Repair in the Era of Large Pre-Trained Language Models. In Proceedings of the 45th International Conference on Software Engineering. 1482\u20131494."},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1607\u20131619","author":"Yang Lin","year":"2024","unstructured":"Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, et\u00a0al. 2024. On the Evaluation of Large Language Models in Unit Test Generation. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 1607\u20131619."},{"key":"e_1_2_1_58_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3505243","article-title":"A Survey on Deep Learning for Software","volume":"54","author":"Yang Yanming","year":"2022","unstructured":"Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022. A Survey on Deep Learning for Software Engineering. Comput. Surveys 54, 10s (2022), 1\u201373.","journal-title":"Engineering. Comput. Surveys"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510149"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the 31th ACM SIGSOFT International Symposium on Software Testing and Analysis. 678\u2013690","author":"Yuan Wei","year":"2022","unstructured":"Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoc\u00a0Viet Hung, Xiaodong Hao, and Hongzhi Yin. 2022. CIRCLE: Continual Repair across Programming Languages. In Proceedings of the 31th ACM SIGSOFT International Symposium on Software Testing and Analysis. 678\u2013690."},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the ACM on Software Engineering 1, FSE","author":"Yuan Zhiqiang","year":"2024","unstructured":"Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703\u20131726."},{"key":"e_1_2_1_62_1","article-title":"A Survey of Learning-based Automated Program Repair","volume":"33","author":"Zhang Quanjun","year":"2024","unstructured":"Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2024. A Survey of Learning-based Automated Program Repair. ACM Trans. Softw. Eng. Methodol. 33, 2 (2024), 55:1\u201355:69.","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"e_1_2_1_63_1","volume-title":"A Systematic Literature Review on Large Language Models for Automated Program Repair. CoRR abs\/2405.01466","author":"Zhang Quanjun","year":"2024","unstructured":"Quanjun Zhang, Chunrong Fang, Yang Xie, Yuxiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A Systematic Literature Review on Large Language Models for Automated Program Repair. CoRR abs\/2405.01466 (2024), arXiv\u20132405."},{"key":"e_1_2_1_64_1","volume-title":"A Survey on Large Language Models for Software Engineering. CoRR abs\/2312.15223","author":"Zhang Quanjun","year":"2023","unstructured":"Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2023. A Survey on Large Language Models for Software Engineering. CoRR abs\/2312.15223 (2023), arXiv\u20132312."},{"key":"e_1_2_1_65_1","doi-asserted-by":"crossref","first-page":"2507","DOI":"10.1109\/TDSC.2023.3308897","article-title":"Pre-Trained Model-Based Automated Software Vulnerability Repair: How Far Are We","volume":"21","author":"Zhang Quanjun","year":"2024","unstructured":"Quanjun Zhang, Chunrong Fang, Bowen Yu, Weisong Sun, Tongke Zhang, and Zhenyu Chen. 2024. Pre-Trained Model-Based Automated Software Vulnerability Repair: How Far Are We? IEEE Transactions on Dependable and Secure Computing 21, 4 (2024), 2507\u20132525.","journal-title":"IEEE Transactions on Dependable and Secure Computing"},{"key":"e_1_2_1_66_1","volume-title":"Gamma: Revisiting Template-Based Automated Program Repair Via Mask Prediction. In 38th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2023","author":"Zhang Quanjun","year":"2023","unstructured":"Quanjun Zhang, Chunrong Fang, Tongke Zhang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. Gamma: Revisiting Template-Based Automated Program Repair Via Mask Prediction. In 38th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 535\u2013547."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3699598"},{"key":"e_1_2_1_68_1","volume-title":"A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair. arXiv preprint arXiv:2310.08879","author":"Zhang Quanjun","year":"2023","unstructured":"Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair. arXiv preprint arXiv:2310.08879 (2023)."},{"key":"e_1_2_1_69_1","volume-title":"2024 IEEE\/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 872\u2013872","author":"Zhou Xin","year":"2024","unstructured":"Xin Zhou, Kisub Kim, Bowen Xu, DongGyun Han, and David Lo. 2024. Out of Sight, Out of Mind: Better Automatic Vulnerability Repair by Broadening Input Ranges and Sources. In 2024 IEEE\/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 872\u2013872."},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313","author":"Zhu Qihao","year":"2024","unstructured":"Qihao Zhu, Qingyuan Liang, Zeyu Sun, Yingfei Xiong, Lu Zhang, and Shengyu Cheng. 2024. GrammarT5: Grammar-Integrated Pretrained Encoder-Decoder Neural Model for Code. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3721128","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T15:32:37Z","timestamp":1740756757000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3721128"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,28]]},"references-count":70,"alternative-id":["10.1145\/3721128"],"URL":"https:\/\/doi.org\/10.1145\/3721128","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,28]]},"assertion":[{"value":"2024-10-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"3721128"}}