{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:59:55Z","timestamp":1782845995898,"version":"3.54.5"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>While Retrieval-Augmented Generation (RAG) is increasingly adopted to ground Large Language Models (LLMs) in software artifacts, the optimal configuration of its components remains an open question for software engineering (SE) tasks. The lack of systematic guidance forces practitioners into costly, ad-hoc experimentation. This paper presents a comprehensive, component-wise empirical study that dissects the RAG pipeline, evaluating over 21 distinct models and methods. Our study systematically isolates and evaluates 4 query processing techniques, 7 retrieval models spanning sparse, dense, and hybrid paradigms, 4 context refinement methods, and 6 distinct generators. We test these components on a suite of 3 core SE tasks: code generation, summarization, and repair. Our empirical findings reveal a crucial insight: the retriever-side components, particularly the choice of the retrieval algorithm, often exert a more significant influence on final system performance than the selection of the generator model. Strikingly, the classic lexical retriever BM25 demonstrates exceptionally robust performance across diverse tasks. Our analysis provides a practical, data-driven roadmap for researchers and practitioners, offering clear guidance on prioritizing optimization efforts when constructing effective RAG systems for software engineering contexts.<\/jats:p>","DOI":"10.1145\/3808190","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"4161-4183","source":"Crossref","is-referenced-by-count":0,"title":["Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Tasks"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-2520-881X","authenticated-orcid":false,"given":"Qiang","family":"Ke","sequence":"first","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8793-5367","authenticated-orcid":false,"given":"Yanjie","family":"Zhao","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3607-8204","authenticated-orcid":false,"given":"Hongjin","family":"Leng","sequence":"additional","affiliation":[{"name":"Xiamen University Malaysia, Sepang, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5035-2206","authenticated-orcid":false,"given":"Shengming","family":"Zhao","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1100-8633","authenticated-orcid":false,"given":"Haoyu","family":"Wang","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL]","author":"Abdin Marah","year":"2024","unstructured":"Marah Abdin, Jyoti Aneja, Hany Awadalla, et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL]"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290353"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2024.09.178"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/llm4code66737.2025.00026"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME64153.2025.00046"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Jianlv Chen Shitao Xiao Peitian Zhang Kun Luo Defu Lian and Zheng Liu. 2024. BGE M3-Embedding: Multi-Lingual Multi-Functionality Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216 [cs.CL]","DOI":"10.18653\/v1\/2024.findings-acl.137"},{"key":"e_1_2_1_7_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Alexis Conneau Kartikay Khandelwal Naman Goyal et al. 2020. Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116 [cs.CL]","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572114"},{"key":"e_1_2_1_10_1","unstructured":"DeepSeek-AI Aixin Liu Bei Feng Bing Xue et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637528"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.ACL-LONG.99"},{"key":"e_1_2_1_14_1","unstructured":"Yunfan Gao Yun Xiong Xinyu Gao Kangxiang Jia Jinliu Pan Yuxi Bi Yi Dai Jiawei Sun Meng Wang and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL]"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2755013"},{"key":"e_1_2_1_16_1","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI]"},{"key":"e_1_2_1_17_1","unstructured":"Wenchao Gu Juntao Chen Yanlin Wang Tianyue Jiang Xingzhe Li Mingwei Liu Xilin Liu Yuchi Ma and Zibin Zheng. 2025. What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond. arXiv:2503.20589 [cs.SE]"},{"key":"e_1_2_1_18_1","volume-title":"Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao.","author":"G\u00fcnther Michael","year":"2024","unstructured":"Michael G\u00fcnther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. 2024. Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents. arXiv:2310.19923 [cs.CL]"},{"key":"e_1_2_1_19_1","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath et al. 2021. Measuring Coding Challenge Competence With APPS. arXiv:2105.09938 [cs.SE]"},{"key":"e_1_2_1_20_1","unstructured":"Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Dawn Song and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874 [cs.LG]"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. arXiv:2007.01282 [cs.CL]","DOI":"10.18653\/v1\/2021.eacl-main.74"},{"key":"e_1_2_1_22_1","first-page":"1","volume-title":"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations, ICLR","author":"Jain Naman","year":"2025","unstructured":"Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations, ICLR 2025. OpenReview.net, Amherst, MA, USA, 1-15."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.91"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2019.2921572"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Vladimir Karpukhin Barlas Oguz Sewon Min et al. 2020. Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906 [cs.CL]","DOI":"10.18653\/v1\/2020.emnlp-main.550"},{"key":"e_1_2_1_26_1","volume-title":"Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela.","author":"Lewis Patrick","year":"2021","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs.CL]"},{"key":"e_1_2_1_27_1","unstructured":"Zehan Li Xin Zhang Yanzhao Zhang Dingkun Long Pengjun Xie and Meishan Zhang. 2023. Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv:2308.03281 [cs.CL]"},{"key":"e_1_2_1_28_1","unstructured":"Ye Liu Rui Meng Shafiq Jot Silvio Savarese Caiming Xiong Yingbo Zhou and Semih Yavuz. 2025. CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval. arXiv:2411.12644 [cs.CL]"},{"key":"e_1_2_1_29_1","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang et al. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. arXiv:2102.04664 [cs.SE]"},{"key":"e_1_2_1_30_1","unstructured":"Xinbei Ma Yeyun Gong Pengcheng He Hai Zhao and Nan Duan. 2023. Query Rewriting for Retrieval-Augmented Large Language Models. arXiv:2305.14283 [cs.CL]"},{"key":"e_1_2_1_31_1","unstructured":"Mayank Mishra Matt Stallone Gaoyuan Zhang et al. 2024. Granite Code Models: A Family of Open Foundation Models for Code Intelligence. arXiv:2405.04324 [cs.AI]"},{"key":"e_1_2_1_32_1","unstructured":"Rodrigo Nogueira and Kyunghyun Cho. 2020. Passage Re-ranking with BERT. arXiv:1901.04085 [cs.IR]"},{"key":"e_1_2_1_33_1","unstructured":"OpenAI. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL]"},{"key":"e_1_2_1_34_1","volume-title":"Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang.","author":"Parvez Md Rizwan","year":"2021","unstructured":"Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval Augmented Code Generation and Summarization. arXiv:2108.11601 [cs.SE]"},{"key":"e_1_2_1_35_1","unstructured":"Shuo Ren Daya Guo Shuai Lu et al. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE]"},{"key":"e_1_2_1_36_1","first-page":"232","volume-title":"SIGIR '94: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval. Springer-Verlag","author":"Robertson S. E.","unstructured":"S. E. Robertson and S. Walker. 1994. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In SIGIR '94: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval. Springer-Verlag, London, UK, 232-241."},{"key":"e_1_2_1_37_1","unstructured":"Jiho Shin Reem Aleithan Hadi Hemmati and Song Wang. 2024. Retrieval-Augmented Test Generation: How Far Are We? arXiv:2409.12682 [cs.SE]"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.431"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2012.6405249"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026526"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.EMNLP-"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Runchu Tian Yining Ye Yujia Qin Xin Cong Yankai Lin Yinxu Pan Yesai Wu Haotian Hui Weichuan Liu Zhiyuan Liu and Maosong Sun. 2024. DebugBench: Evaluating Debugging Capability of Large Language Models. arXiv:2401.04621 [cs.SE]","DOI":"10.18653\/v1\/2024.findings-acl.247"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985907"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549113"},{"key":"e_1_2_1_45_1","unstructured":"Liang Wang Nan Yang Xiaolong Huang Binxing Jiao et al. 2024. Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv:2212.03533 [cs.CL]"},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Liang Wang Nan Yang Xiaolong Huang Linjun Yang Rangan Majumder and Furu Wei. 2024. Improving Text Embeddings with Large Language Models. arXiv:2401.00368 [cs.CL]","DOI":"10.18653\/v1\/2024.acl-long.642"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Xiaohua Wang Zhenghua Wang Xuan Gao Feiran Zhang Yixin Wu Zhibo Xu Tianyuan Shi Zhengyuan Wang Shizheng Li Qi Qian Ruicheng Yin Changze Lv Xiaoqing Zheng and Xuanjing Huang. 2024. Searching for Best Practices in Retrieval-Augmented Generation. arXiv:2407.01219 [cs.CL]","DOI":"10.18653\/v1\/2024.emnlp-main.981"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Yuan Wang Xuyang Wu Hsin-Tai Wu Zhiqiang Tao and Yi Fang. 2024. Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers. arXiv:2404.03192 [cs.IR]","DOI":"10.18653\/v1\/2024.naacl-long.319"},{"key":"e_1_2_1_49_1","first-page":"24824","volume-title":"Oh (Eds.)","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., Red Hook, NY, USA, 24824-24837."},{"key":"e_1_2_1_50_1","volume-title":"RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. arXiv:2310.04408 [cs.CL]","author":"Xu Fangyuan","year":"2023","unstructured":"Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. arXiv:2310.04408 [cs.CL]"},{"key":"e_1_2_1_51_1","unstructured":"An Yang Baosong Yang Beichen Zhang et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL]"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3717061"},{"key":"e_1_2_1_53_1","unstructured":"Yanzhao Zhang Mingxin Li Dingkun Long Xin Zhang Huan Lin Baosong Yang Pengjun Xie An Yang Dayiheng Liu Junyang Lin Fei Huang and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176 [cs.CL]"},{"key":"e_1_2_1_54_1","unstructured":"Shuyan Zhou Uri Alon Frank F. Xu Zhiruo Wang Zhengbao Jiang and Graham Neubig. 2023. DocPrompting: Generating Code by Retrieving the Docs. arXiv:2207.05987 [cs.SE]"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-26409-2_5"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808190","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:02:27Z","timestamp":1782842547000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808190"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":55,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3808190"],"URL":"https:\/\/doi.org\/10.1145\/3808190","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}