{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T02:37:00Z","timestamp":1774579020318,"version":"3.50.1"},"reference-count":160,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,5,24]],"date-time":"2025-05-24T00:00:00Z","timestamp":1748044800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001321","name":"National Research Foundation","doi-asserted-by":"crossref","award":["NRF-NRFI08-2022-0002"],"award-info":[{"award-number":["NRF-NRFI08-2022-0002"]}],"id":[{"id":"10.13039\/501100001321","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions.<\/jats:p>\n          <jats:p>This article aims to redirect the focus of the research community toward the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the article presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The article then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the article aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering.<\/jats:p>","DOI":"10.1145\/3708525","type":"journal-article","created":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T13:58:06Z","timestamp":1734703086000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0799-5018","authenticated-orcid":false,"given":"Jieke","family":"Shi","sequence":"first","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5938-1918","authenticated-orcid":false,"given":"Zhou","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4367-7201","authenticated-orcid":false,"given":"David","family":"Lo","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, Singapore Management University, Singapore, Singapore"}]}],"member":"320","published-online":{"date-parts":[[2025,5,24]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559555"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639183"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644815.3644967"},{"key":"e_1_3_1_6_2","unstructured":"Gareth Ari Aye and Gail E. Kaiser. 2020. Sequence model design for code completion in the modern IDE. arXiv:2004.05249. Retrieved from https:\/\/arxiv.org\/abs\/2004.05249"},{"key":"e_1_3_1_7_2","unstructured":"Shamil Ayupov and Nadezhda Chirkova. 2022. Parameter-efficient finetuning of transformers for source code. arXiv:2212.05901. Retrieved from https:\/\/arxiv.org\/abs\/2212.05901"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclepro.2017.12.239"},{"key":"e_1_3_1_9_2","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/978-3-031-46002-9_23","volume-title":"Bridging the Gap between AI and Reality","author":"Belzner Lenz","year":"2024","unstructured":"Lenz Belzner, Thomas Gabor, and Martin Wirsing. 2024. Large language model assisted software engineering: Prospects, challenges, and a case study. In Bridging the Gap between AI and Reality, Bernhard Steffen (Ed.), Springer Nature Switzerland, Cham, 355\u2013374."},{"key":"e_1_3_1_10_2","first-page":"1877","volume-title":"Advances in Neural Information Processing SystemsCurran Associates, Inc","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877\u20131901. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf"},{"key":"e_1_3_1_11_2","unstructured":"Coral Calero F\u00e9lix O. Garc\u00eda Gabriel Alberto Garc\u00eda-Mireles M. \u00c1ngeles Moraga and Aurora Vizca\u00edno. 2024. Addressing Sustainability-IN Software Challenges. arXiv:2406.07380.Retrieved from https:\/\/arxiv.org\/abs\/2406.07380"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549162"},{"key":"e_1_3_1_13_2","unstructured":"Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv:2302.01318.Retrieved from https:\/\/arxiv.org\/abs\/2302.01318"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639085"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00110"},{"key":"e_1_3_1_16_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374.s Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3365438.3410949"},{"key":"e_1_3_1_18_2","unstructured":"Nicole Choi. 2024. Customizing and fine-tuning LLMs: What you need to know \u2014 github.blog. Retrieved March 28 2024 from https:\/\/github.blog\/2024-02-28-customizing-and-fine-tuning-llms-what-you-need-to-know\/"},{"key":"e_1_3_1_19_2","unstructured":"Google Cloud. 2023. Sec-PaLM2. Retrieved March 26 2024 from https:\/\/console.cloud.google.com\/vertex-ai\/publishers\/google\/model-garden\/sec-palm"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3664646.3664773"},{"key":"e_1_3_1_21_2","doi-asserted-by":"crossref","unstructured":"Lu\u00eds Cruz Xavier Franch Gutierrez and Silverio Mart\u00ednez-Fern\u00e1ndez. 2024. Innovating for tomorrow: The convergence of SE and green AI. arXiv:2406.18142. Retrieved from https:\/\/arxiv.org\/abs\/2406.18142","DOI":"10.1145\/3712007"},{"key":"e_1_3_1_22_2","unstructured":"Cursor. 2023. Inference characteristics of llama-2. Retrieved September 29 2024 from https:\/\/www.cursor.com\/blog\/llama-inference"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.joule.2023.09.004"},{"key":"e_1_3_1_24_2","first-page":"6861","volume-title":"Proceedings of the 33rd USENIX Security Symposium (USENIX Security \u201924)","author":"Debenedetti Edoardo","year":"2024","unstructured":"Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A. Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tram\u00e8r. 2024. Privacy Side channels in machine learning systems. In Proceedings of the 33rd USENIX Security Symposium (USENIX Security \u201924). USENIX Association, Philadelphia, PA, 6861\u20136848. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity24\/presentation\/debenedetti"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644815.3644962"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPCCC59175.2023.10253886"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671470"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608137"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_32_2","doi-asserted-by":"crossref","unstructured":"Marjan Ghazvininejad Omer Levy Yinhan Liu and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. arXiv:1904.09324. Retrieved from https:\/\/arxiv.org\/abs\/1904.09324","DOI":"10.18653\/v1\/D19-1633"},{"key":"e_1_3_1_33_2","unstructured":"GitHub. 2023. GitHub Copilot community. Retrieved October 03 2023 from https:\/\/github.com\/orgs\/community\/discussions\/categories\/copilot?discussions_q=category%3ACopilot+network"},{"key":"e_1_3_1_34_2","unstructured":"GitHub. 2023. GitHub Copilot \\({\\cdot}\\) your AI pair programmer. Retrieved March 22 2024 from https:\/\/github.com\/features\/copilot\/"},{"key":"e_1_3_1_35_2","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1007\/978-3-031-71139-8_5","volume-title":"Systems, Software and Services Process Improvement","author":"G\u00f6rmez Muhammet K\u00fcr\u015fat","year":"2024","unstructured":"Muhammet K\u00fcr\u015fat G\u00f6rmez, Murat Yi\u0307lmaz, and Paul M. Clarke. 2024. Large language models for software engineering: A systematic mapping study. In Systems, Software and Services Process Improvement. Murat Yilmaz, Paul Clarke, Andreas Riel, Richard Messnarz, Christian Greiner, and Thomas Peisl (Eds.), Springer Nature Switzerland, Cham, 64\u201379."},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616369"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616304"},{"key":"e_1_3_1_38_2","volume-title":"International Conference on Learning Representations","author":"Gu Jiatao","year":"2018","unstructured":"Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=B1l8BtlCb"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.181"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2024.01.033"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.499"},{"key":"e_1_3_1_42_2","volume-title":"International Conference on Learning Representations","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2021. GraphCode{BERT}: Pre-training code representations with data flow. In International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=jLoC4ez43PZ"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680343"},{"key":"e_1_3_1_44_2","unstructured":"Matt Hamblen. 2023. Update: ChatGPT runs 10K nvidia training GPUs with potential for thousands more. Retrieved April 02 2024 from https:\/\/www.fierceelectronics.com\/sensors\/chatgpt-runs-10k-nvidia-training-gpus-potential-thousands-more"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3635711"},{"key":"e_1_3_1_46_2","unstructured":"D\u00e1vid Hidv\u00e9gi Khashayar Etemadi Sofia Bobadilla and Martin Monperrus. 2024. CigaR: Cost-efficient program repair with LLMs. arXiv:2402.06598. Retrieved from https:\/\/arxiv.org\/abs\/2402.06598"},{"key":"e_1_3_1_47_2","first-page":"30016","volume-title":"Advances in Neural Information Processing SystemsCurran Associates, Inc","volume":"35","author":"Hoffmann Jordan","year":"2022","unstructured":"Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 30016\u201330030. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591687"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_1_50_2","first-page":"15500","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING \u201924)","author":"Hu Fan","year":"2024","unstructured":"Fan Hu, Yanlin Wang, Lun Du, Hongyu Zhang, Dongmei Zhang, and Xirong Li. 2024. Tackling long code search with splitting, encoding, and aggregating. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING \u201924). Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.), ELRA and ICCL, Torino, Italia, 15500\u201315510. Retrieved from https:\/\/aclanthology.org\/2024.lrec-main.1347"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3376964"},{"key":"e_1_3_1_52_2","unstructured":"Dong Huang Yuhao Qing Weiyi Shang Heming Cui and Jie M. Zhang. 2024. EffiBench: Benchmarking the efficiency of automatically generated code. arXiv:2402.02037.Retrieved from https:\/\/arxiv.org\/abs\/2402.02037"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660818"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644815.3644943"},{"key":"e_1_3_1_55_2","unstructured":"International Energy Agency (IEA). 2024. Energy system of Denmark. Retrieved September 25 2024 from https:\/\/www.iea.org\/countries\/denmark"},{"key":"e_1_3_1_56_2","doi-asserted-by":"crossref","unstructured":"Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. arXiv:2007.01282.Retrieved from https:\/\/arxiv.org\/abs\/2007.01282","DOI":"10.18653\/v1\/2021.eacl-main.74"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639475.3640111"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00125"},{"key":"e_1_3_1_59_2","unstructured":"Chao Jin Zili Zhang Xuanlin Jiang Fangyue Liu Xin Liu Xuanzhe Liu and Xin Jin. 2024. RAGCache: Efficient knowledge caching for retrieval-augmented generation. arXiv:2404.12457.Retrieved from https:\/\/arxiv.org\/abs\/2404.12457"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613892"},{"key":"e_1_3_1_61_2","unstructured":"Eirini Kalliamvakou. 2022. Research: Quantifying GitHub copilot\u2019s impact on developer productivity and happiness. Retrieved September 26 2024 from https:\/\/github.blog\/news-insights\/research\/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness\/"},{"key":"e_1_3_1_62_2","series-title":"Proceedings of Machine Learning Research","first-page":"5110","volume-title":"Proceedings of the 37th International Conference on Machine Learning","volume":"119","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). Hal Daum\u00e9 III and Aarti Singh (Eds.), PMLR, 5110\u20135121. Retrieved from https:\/\/proceedings.mlr.press\/v119\/kanade20a.html"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644815.3644942"},{"key":"e_1_3_1_64_2","unstructured":"Ayush Kaushal Tejas Vaidhya and Irina Rish. 2023. LORD: Low rank decomposition of monolingual code LLMs for one-shot compression. arXiv:2309.14021.Retrieved from https:\/\/arxiv.org\/abs\/2309.14021"},{"key":"e_1_3_1_65_2","series-title":"Proceedings of Machine Learning Research","first-page":"16639","volume-title":"Proceedings of the 40th International Conference on Machine Learning","volume":"202","author":"Kim Taebum","year":"2023","unstructured":"Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun. 2023. BPipe: Memory-balanced pipeline parallelism for training large language models. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), PMLR, 16639\u201316653. Retrieved from https:\/\/proceedings.mlr.press\/v202\/kim23l.html"},{"key":"e_1_3_1_66_2","unstructured":"Jennifer L. 2023. Meta to buy almost 7 million Carbon credits from aspiration \u2014 carboncredits.com. Retrieved March 28 2024 from https:\/\/carboncredits.com\/meta-to-buy-almost-7-million-carbon-credits-from-aspiration\/"},{"key":"e_1_3_1_67_2","unstructured":"Kif Leswing. 2023. ChatGPT and generative AI are booming but the costs can be extraordinary. Retrieved April 02 2024 from https:\/\/www.cnbc.com\/2023\/03\/13\/chatgpt-and-generative-ai-are-booming-but-at-a-very-expensive-price.html"},{"key":"e_1_3_1_68_2","series-title":"Proceedings of Machine Learning Research","first-page":"19274","volume-title":"Proceedings of the 40th International Conference on Machine Learning","volume":"202","author":"Leviathan Yaniv","year":"2023","unstructured":"Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), PMLR, 19274\u201319286. Retrieved from https:\/\/proceedings.mlr.press\/v202\/leviathan23a.html"},{"key":"e_1_3_1_69_2","first-page":"9459","volume-title":"Advances in Neural Information Processing SystemsCurran Associates, Inc","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9459\u20139474. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/6b493230205f780e1bc26945df7481e5-Paper.pdf"},{"key":"e_1_3_1_70_2","unstructured":"Baolin Li Yankai Jiang Vijay Gadepally and Devesh Tiwari. 2024. Toward sustainable GenAI using generation directives for carbon-friendly large language model inference. arXiv:2403.12900.Retrieved from https:\/\/arxiv.org\/abs\/2403.12900"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695066"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643916.3644434"},{"key":"e_1_3_1_73_2","unstructured":"Pengfei Li Jianyi Yang Mohammad A. Islam and Shaolei Ren. 2023b. Making AI less \u201cthirsty\u201d: Uncovering and addressing the secret water footprint of AI models. arXiv:2304.03271.Retrieved from https:\/\/arxiv.org\/abs\/2304.03271"},{"key":"e_1_3_1_74_2","unstructured":"Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim et al. 2023. StarCoder: May the source be with you! arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.187"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00125"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER60148.2024.00055"},{"key":"e_1_3_1_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643674"},{"key":"e_1_3_1_80_2","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. Retrieved from https:\/\/arxiv.org\/abs\/1907.11692"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00010"},{"key":"e_1_3_1_82_2","unstructured":"Anton Lozhkov Raymond Li Loubna Ben Allal Federico Cassano Joel Lamy-Poirier Nouamane Tazi Ao Tang Dmytro Pykhtar Jiawei Liu Yuxiang Wei et al. 2024. StarCoder 2 and the Stack v2: The next generation. arXiv:2402.19173. Retrieved from https:\/\/arxiv.org\/abs\/2402.19173"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE59848.2023.00026"},{"key":"e_1_3_1_84_2","volume-title":"Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Retrieved from https:\/\/openreview.net\/forum?id=6lE4dQXaUcb"},{"issue":"253","key":"e_1_3_1_85_2","first-page":"1","article-title":"Estimating the carbon footprint of BLOOM, a 176B parameter language model","volume":"24","author":"Luccioni Alexandra Sasha","year":"2023","unstructured":"Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2023. Estimating the carbon footprint of BLOOM, a 176B parameter language model. Journal of Machine Learning Research 24, 253 (2023), 1\u201315. Retrieved from http:\/\/jmlr.org\/papers\/v24\/23-0069.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_86_2","unstructured":"Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Hongyi Jin Tianqi Chen and Zhihao Jia. 2023. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv:2312.15234. Retrieved from https:\/\/arxiv.org\/abs\/2312.15234"},{"key":"e_1_3_1_87_2","unstructured":"Erik Nijkamp Hiroaki Hayashi Caiming Xiong Silvio Savarese and Yingbo Zhou. 2023. CodeGen2: Lessons for training LLMs on programming and natural languages. arXiv:2305.02309. Retrieved from https:\/\/arxiv.org\/abs\/2305.02309"},{"key":"e_1_3_1_88_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Nijkamp Erik","year":"2023","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. In Proceedings of the 11th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=iaYcJKpY2B_"},{"key":"e_1_3_1_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510096"},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608136"},{"key":"e_1_3_1_91_2","first-page":"2133","volume-title":"Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923)","author":"Niu Liang","year":"2023","unstructured":"Liang Niu, Shujaat Mirza, Zayd Maradni, and Christina P\u00f6pper. 2023. CodexLeaks: Privacy leaks from code generation language models in GitHub copilot. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923). USENIX Association, Anaheim, CA, 2133\u20132150. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity23\/presentation\/niu"},{"key":"e_1_3_1_92_2","unstructured":"OpenAI. 2024. GPT-4 technical report. arXiv:2303.08774.Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP54263.2024.00152"},{"key":"e_1_3_1_94_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.232"},{"key":"e_1_3_1_95_2","unstructured":"David Patterson Joseph Gonzalez Quoc Le Chen Liang Lluis-Miquel Munguia Daniel Rothchild David So Maud Texier and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv:2104.10350.Retrieved from https:\/\/arxiv.org\/abs\/2104.10350"},{"key":"e_1_3_1_96_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.nlp4prog-1.5"},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW60843.2023.00058"},{"key":"e_1_3_1_98_2","unstructured":"Mario Rodriguez. 2023. Research: Quantifying GitHub copilot\u2019s impact on code quality. Retrieved September 26 2024 from https:\/\/github.blog\/news-insights\/research\/research-quantifying-github-copilots-impact-on-code-quality\/"},{"key":"e_1_3_1_99_2","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Romain Sauvestre Tal Remez et al. 2024. Code Llama: Open foundation models for code. arXiv:2308.12950. [cs.CL] Retrieved from https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_1_100_2","unstructured":"Mootez Saad Jos\u00e9 Antonio Hern\u00e1ndez L\u00f3pez Boqi Chen D\u00e1niel Varr\u00f3 and Tushar Sharma. 2024. ALPINE: An adaptive language-agnostic pruning method for language models for code. arXiv:2407.04147. Retrieved from https:\/\/arxiv.org\/abs\/2407.04147"},{"key":"e_1_3_1_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC58863.2023.10363447"},{"key":"e_1_3_1_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3334955"},{"key":"e_1_3_1_103_2","unstructured":"Google Cloud Security. 2023. Introducing VirusTotal code insight: Empowering threat analysis with generative AI. Retrieved March 26 2024 from https:\/\/blog.virustotal.com\/2023\/04\/introducing-virustotal-code-insight.html"},{"key":"e_1_3_1_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3609437.3609438"},{"key":"e_1_3_1_105_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598036"},{"key":"e_1_3_1_106_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00130"},{"key":"e_1_3_1_107_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639475.3640097"},{"key":"e_1_3_1_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556964"},{"key":"e_1_3_1_109_2","unstructured":"Jasvinder Singh. 2023. What is the energy consumption of AI enabled large language modelling (LLM\u2019S) types of searches? \u2014 linkedin.com. Retrieved March 27 2024 from https:\/\/www.linkedin.com\/pulse\/what-energy-consumption-ai-enabled-large-language-modelling-singh"},{"key":"e_1_3_1_110_2","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1007\/978-3-030-64437-6_16","volume-title":"Programming Languages and Systems","author":"Steinh\u00f6fel Dominic","year":"2020","unstructured":"Dominic Steinh\u00f6fel. 2020. REFINITY to model and Prove program transformation rules. In Programming Languages and Systems. Bruno C. D. S. Oliveira (Ed.), Springer International Publishing, Cham, 311\u2013319."},{"key":"e_1_3_1_111_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-024-00421-4"},{"key":"e_1_3_1_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639120"},{"key":"e_1_3_1_113_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion58688.2023.00089"},{"key":"e_1_3_1_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/3688831"},{"key":"e_1_3_1_115_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680347"},{"key":"e_1_3_1_116_2","unstructured":"Google Sustainability. 2023. Aiming to achieve net-zero emissions. Retrieved March 28 2024 from https:\/\/sustainability.google\/operating-sustainably\/net-zero-carbon\/"},{"key":"e_1_3_1_117_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_3_1_118_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00045"},{"key":"e_1_3_1_119_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.443"},{"key":"e_1_3_1_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564625.3567985"},{"key":"e_1_3_1_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3503222.3507764"},{"key":"e_1_3_1_122_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. LLaMA: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_1_123_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_1_124_2","unstructured":"Shubham Ugare Tarun Suresh Hangoo Kang Sasa Misailovic and Gagandeep Singh. 2024. SynCode: LLM generation with grammar augmentation. arXiv:2403.01632. Retrieved from https:\/\/arxiv.org\/abs\/2403.01632"},{"key":"e_1_3_1_125_2","doi-asserted-by":"crossref","unstructured":"Tina Vartziotis Ippolyti Dellatolas George Dasoulas Maximilian Schmidt Florian Schneider Tim Hoffmann Sotirios Kotsopoulos and Michael Keckeisen. 2024. Learn to code sustainably: An empirical study on LLM-based green code generation. arXiv:2403.03344. Retrieved from https:\/\/arxiv.org\/abs\/2403.03344","DOI":"10.1145\/3643795.3648394"},{"key":"e_1_3_1_126_2","volume-title":"Advances in Neural Information Processing Systems","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"e_1_3_1_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643762"},{"key":"e_1_3_1_128_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00013"},{"key":"e_1_3_1_129_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510062"},{"key":"e_1_3_1_130_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.xinn.2021.100180"},{"key":"e_1_3_1_131_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_1_132_2","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Wang Yue","year":"2023","unstructured":"Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven Hoi. 2023. CodeT5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Retrieved from https:\/\/openreview.net\/forum?id=uu6Oq7MN7g"},{"key":"e_1_3_1_133_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643753"},{"key":"e_1_3_1_134_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_1_135_2","unstructured":"Zora Zhiruo Wang Akari Asai Xinyan Velocity Yu Frank F. Xu Yiqing Xie Graham Neubig and Daniel Fried. 2024. CodeRAG-Bench: Can retrieval augment code generation? arXiv:2406.14497. Retrieved from https:\/\/arxiv.org\/abs\/2406.14497"},{"key":"e_1_3_1_136_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616302"},{"key":"e_1_3_1_137_2","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Wei Yuxiang","year":"2024","unstructured":"Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with OSS-instruct. In Proceedings of the 41st International Conference on Machine Learning. Retrieved from https:\/\/openreview.net\/forum?id=XUeoOBid3x"},{"key":"e_1_3_1_138_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616271"},{"key":"e_1_3_1_139_2","unstructured":"Sarah Wells. 2023. Generative AI\u2019s energy problem today is foundational. Retrieved March 28 2024 from https:\/\/spectrum.ieee.org\/ai-energy-consumption"},{"key":"e_1_3_1_140_2","doi-asserted-by":"crossref","unstructured":"Martin Weyssow Xin Zhou Kisub Kim David Lo and Houari Sahraoui. 2024. Exploring parameter-efficient fine-tuning techniques for code generation with large language models. arXiv:2308.10462. Retrieved from https:\/\/arxiv.org\/abs\/2308.10462","DOI":"10.1145\/3714461"},{"key":"e_1_3_1_141_2","unstructured":"Wikipedia. 2024. List of nvidia graphics processing units. Retrieved September 29 2024 from https:\/\/en.wikipedia.org\/wiki\/List_of_Nvidia_graphics_processing_units"},{"key":"e_1_3_1_142_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_1_143_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_1_144_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649506"},{"key":"e_1_3_1_145_2","doi-asserted-by":"crossref","unstructured":"Zhou Yang Zhensu Sun Terry Zhuo Yue Premkumar Devanbu and David Lo. 2024. Robustness security privacy explainability efficiency and usability of large language models for code. arXiv:2403.07506. Retrieved from https:\/\/arxiv.org\/abs\/2403.07506","DOI":"10.1145\/3731753"},{"key":"e_1_3_1_146_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639074"},{"key":"e_1_3_1_147_2","unstructured":"Deborah Yao. 2023. Microsoft\u2019s GitHub copilot loses $20 a month per user. Retrieved March 28 2024 from https:\/\/aibusiness.com\/nlp\/github-copilot-loses-20-a-month-per-user"},{"key":"e_1_3_1_148_2","volume-title":"International Conference on Learning Representations","author":"You Yang","year":"2020","unstructured":"Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=Syx4wnEtvH"},{"key":"e_1_3_1_149_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623316"},{"key":"e_1_3_1_150_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644815.3644966"},{"key":"e_1_3_1_151_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556955"},{"key":"e_1_3_1_152_2","unstructured":"Quanjun Zhang Chunrong Fang Yang Xie Yaxin Zhang Yun Yang Weisong Sun Shengcheng Yu and Zhenyu Chen. 2024. A survey on large language models for software engineering. arXiv:2312.15223. Retrieved from https:\/\/arxiv.org\/abs\/2312.15223"},{"key":"e_1_3_1_153_2","unstructured":"Ting Zhang Ivana Clairine Irsan Ferdian Thung and David Lo. 2024. CUPID: Leveraging ChatGPT for more accurate duplicate bug report detection. arXiv:2308.10022. Retrieved from https:\/\/arxiv.org\/abs\/2308.10022"},{"key":"e_1_3_1_154_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549094"},{"key":"e_1_3_1_155_2","unstructured":"Jiawei Zhao Zhenyu Zhang Beidi Chen Zhangyang Wang Anima Anandkumar and Yuandong Tian. 2024. GaLore: Memory-efficient LLM training by gradient low-rank projection. arXiv:2403.03507. Retrieved from https:\/\/arxiv.org\/abs\/2403.03507"},{"key":"e_1_3_1_156_2","unstructured":"Zibin Zheng Kaiwen Ning Yanlin Wang Jingwen Zhang Dewu Zheng Mingxi Ye and Jiachi Chen. 2024. A survey of large language models for code: Evolution benchmarking and future trends. arXiv:2311.10372. Retrieved from https:\/\/arxiv.org\/abs\/2311.10372"},{"key":"e_1_3_1_157_2","unstructured":"Fan Zhou Zengzhi Wang Qian Liu Junlong Li and Pengfei Liu. 2024. Programming every example: Lifting pre-training data quality like experts at scale. arXiv:2409.17115. Retrieved from https:\/\/arxiv.org\/abs\/2409.17115"},{"key":"e_1_3_1_158_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58846.2023.00028"},{"key":"e_1_3_1_159_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639762"},{"key":"e_1_3_1_160_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639125"},{"key":"e_1_3_1_161_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3558959"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708525","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3708525","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:46Z","timestamp":1750295866000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708525"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,24]]},"references-count":160,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3708525"],"URL":"https:\/\/doi.org\/10.1145\/3708525","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,24]]},"assertion":[{"value":"2024-04-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}