{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:43:58Z","timestamp":1776113038046,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":46,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,4,14]],"date-time":"2024-04-14T00:00:00Z","timestamp":1713052800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,4,14]]},"DOI":"10.1145\/3639476.3639764","type":"proceedings-article","created":{"date-parts":[[2024,5,24]],"date-time":"2024-05-24T15:15:01Z","timestamp":1716563701000},"page":"102-106","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":70,"title":["Breaking the Silence: the Threats of Using LLMs in Software Engineering"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2230-9351","authenticated-orcid":false,"given":"June","family":"Sallou","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1996-6134","authenticated-orcid":false,"given":"Thomas","family":"Durieux","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7395-3588","authenticated-orcid":false,"given":"Annibale","family":"Panichella","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2024,5,24]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Hugging Face - The AI community building the future. https:\/\/huggingface.co [Online","year":"2023","unstructured":"2023. Hugging Face - The AI community building the future. https:\/\/huggingface.co [Online; accessed 11. Sept. 2023]."},{"key":"e_1_3_2_1_2_1","volume-title":"LeetCode - The World's Leading Online Programming Learning Platform. https:\/\/leetcode.com [Online","year":"2023","unstructured":"2023. LeetCode - The World's Leading Online Programming Learning Platform. https:\/\/leetcode.com [Online; accessed 12. Sept. 2023]."},{"key":"e_1_3_2_1_3_1","volume-title":"https:\/\/zenodo.org [Online","year":"2023","unstructured":"2023. Zenodo. https:\/\/zenodo.org [Online; accessed 11. Sept. 2023]."},{"key":"e_1_3_2_1_4_1","volume-title":"Muhammad Waseem Anwar, Farooque Azam, and Bilal Maqbool.","author":"Ain Qurat Ul","year":"2019","unstructured":"Qurat Ul Ain, Wasi Haider Butt, Muhammad Waseem Anwar, Farooque Azam, and Bilal Maqbool. 2019. A systematic review on code clone detection. IEEE access 7 (2019), 86121--86144."},{"key":"e_1_3_2_1_5_1","volume-title":"2023 IEEE\/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE).","author":"Al-Kaswan Ali","year":"2023","unstructured":"Ali Al-Kaswan and Maliheh Izadi. 2023. The (ab)use of Open Source Code to Train Large Language Models. In 2023 IEEE\/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE)."},{"key":"e_1_3_2_1_6_1","volume-title":"A3Test: Assertion-Augmented Automated Test Case Generation. arXiv preprint arXiv:2302.10352","author":"Alagarsamy Saranya","year":"2023","unstructured":"Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2023. A3Test: Assertion-Augmented Automated Test Case Generation. arXiv preprint arXiv:2302.10352 (2023)."},{"key":"e_1_3_2_1_7_1","unstructured":"Ebtesam Almazrouei Hamza Alobeidli Abdulaziz Alshamsi Alessandro Cappelli Ruxandra Cojocaru Maitha Alhammadi Mazzotta Daniele Daniel Heslow Julien Launay Quentin Malartic Badreddine Noune Baptiste Pannier and Guilherme Penedo. 2023. The Falcon Series of Language Models: Towards Open Frontier Models. (2023)."},{"key":"e_1_3_2_1_8_1","volume-title":"Proc. of the Genetic and Evolutionary Computation Conference. 1490--1498","author":"Applis Leonhard","year":"2023","unstructured":"Leonhard Applis, Annibale Panichella, and Ruben Marang. 2023. Searching for Quality: Genetic Algorithms and Metamorphic Testing for Software Engineering ML. In Proc. of the Genetic and Evolutionary Computation Conference. 1490--1498."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678706"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_2_1_11_1","volume-title":"Is github's copilot as bad as humans at introducing vulnerabilities in code? arXiv preprint arXiv:2204.04741","author":"Asare Owura","year":"2022","unstructured":"Owura Asare, Meiyappan Nagappan, and N Asokan. 2022. Is github's copilot as bad as humans at introducing vulnerabilities in code? arXiv preprint arXiv:2204.04741 (2022)."},{"key":"e_1_3_2_1_12_1","volume-title":"Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al.","author":"Athiwaratkun Ben","year":"2022","unstructured":"Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868 (2022)."},{"key":"e_1_3_2_1_13_1","unstructured":"Authors. 2023. https:\/\/github.com\/LLM4SE\/obfuscated-ChatGPT-experiments"},{"key":"e_1_3_2_1_14_1","volume-title":"How is ChatGPT's behavior changing over time? arXiv preprint arXiv:2307.09009 (July","author":"Chen Lingjiao","year":"2023","unstructured":"Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is ChatGPT's behavior changing over time? arXiv preprint arXiv:2307.09009 (July 2023). arXiv:2307.09009"},{"key":"e_1_3_2_1_15_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510457.3513081"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387445"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","first-page":"e1838","DOI":"10.1002\/stvr.1838","article-title":"JUGE: An infrastructure for benchmarking Java unit test generators","volume":"33","author":"Devroey Xavier","year":"2023","unstructured":"Xavier Devroey, Alessio Gambi, Juan Pablo Galeotti, Ren\u00e9 Just, Fitsum Kifetew, Annibale Panichella, and Sebastiano Panichella. 2023. JUGE: An infrastructure for benchmarking Java unit test generators. Software Testing, Verification and Reliability 33, 3 (2023), e1838.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.2196\/48305"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00128"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_3_2_1_22_1","volume-title":"Training Data Leakage Analysis in Language Models. (February","author":"Inan Huseyin Atahan","year":"2021","unstructured":"Huseyin Atahan Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor R\u00fchle, James Withers, and Robert Sim. 2021. Training Data Leakage Analysis in Language Models. (February 2021). https:\/\/www.microsoft.com\/en-us\/research\/publication\/training-data-leakage-analysis-in-language-models\/"},{"key":"e_1_3_2_1_23_1","volume-title":"Stupid Bugs. arXiv preprint arXiv:2303.11455","author":"Jesse Kevin","year":"2023","unstructured":"Kevin Jesse, Toufique Ahmed, Premkumar T Devanbu, and Emily Morgan. 2023. Large Language Models and Simple, Stupid Bugs. arXiv preprint arXiv:2303.11455 (2023)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2628055"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.02684"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00085"},{"key":"e_1_3_2_1_27_1","volume-title":"Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474","author":"Nijkamp Erik","year":"2022","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)."},{"key":"e_1_3_2_1_28_1","volume-title":"https:\/\/openai.com\/ Accessed on September 14th","author":"AI.","year":"2023","unstructured":"OpenAI. 2023. OpenAI. https:\/\/openai.com\/ Accessed on September 14th, 2023."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3248401"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102604"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/SBST52555.2021.00011"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP46215.2023.10179420"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Luiza Pozzobon Beyza Ermis Patrick Lewis and Sara Hooker. 2023. On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research. arXiv:2304.12397 [cs.CL]","DOI":"10.18653\/v1\/2023.emnlp-main.472"},{"key":"e_1_3_2_1_34_1","volume-title":"Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes.","author":"Siddiq Mohammed Latif","year":"2023","unstructured":"Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. 2023. Exploring the Effectiveness of Large Language Models in Generating Unit Tests. arXiv preprint arXiv:2305.00418 (2023)."},{"key":"e_1_3_2_1_35_1","volume-title":"ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation. arXiv preprint arXiv:2307.00588","author":"Tang Yutian","year":"2023","unstructured":"Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2023. ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation. arXiv preprint arXiv:2307.00588 (2023)."},{"key":"e_1_3_2_1_36_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_2_1_37_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_3_2_1_38_1","volume-title":"Shao Kun Deng, and Neel Sundaresan","author":"Tufano Michele","year":"2020","unstructured":"Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020)."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1507"},{"key":"e_1_3_2_1_40_1","volume-title":"Large Language Models in Fault Localisation. arXiv preprint arXiv:2308.15276","author":"Wu Yonghao","year":"2023","unstructured":"Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large Language Models in Fault Localisation. arXiv preprint arXiv:2308.15276 (2023)."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_2_1_42_1","volume-title":"Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385","author":"Xia Chunqiu Steven","year":"2023","unstructured":"Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023)."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_2_1_44_1","unstructured":"Wentao Ye Mingfeng Ou Tianyi Li Xuetao Ma Yifan Yanggong Sai Wu Jie Fu Gang Chen Junbo Zhao et al. 2023. Assessing Hidden Risks of LLMs: An Empirical Study on Robustness Consistency and Credibility. arXiv preprint arXiv:2305.10235 (2023)."},{"key":"e_1_3_2_1_45_1","volume-title":"Proc. of the ACM on Programming Languages 4, OOPSLA","author":"Yefet Noam","year":"2020","unstructured":"Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. Proc. of the ACM on Programming Languages 4, OOPSLA (2020), 1--30."},{"key":"e_1_3_2_1_46_1","unstructured":"Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu Tingchen Fu Xinting Huang Enbo Zhao Yu Zhang Yulong Chen et al. 2023. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint arXiv:2309.01219 (2023)."}],"event":{"name":"ICSE-NIER'24: 2024 ACM\/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results","location":"Lisbon Portugal","acronym":"ICSE-NIER'24","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS","Faculty of Engineering of University of Porto"]},"container-title":["Proceedings of the 2024 ACM\/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639476.3639764","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639476.3639764","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:53:38Z","timestamp":1750287218000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639476.3639764"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,14]]},"references-count":46,"alternative-id":["10.1145\/3639476.3639764","10.1145\/3639476"],"URL":"https:\/\/doi.org\/10.1145\/3639476.3639764","relation":{},"subject":[],"published":{"date-parts":[[2024,4,14]]},"assertion":[{"value":"2024-05-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}