{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T17:15:49Z","timestamp":1783444549611,"version":"3.54.6"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"4","funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2024YFB4506400"],"award-info":[{"award-number":["2024YFB4506400"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"name":"CCF-Huawei Populus Grove Fund."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,4,30]]},"abstract":"<jats:p>Large language models achieve promising results in code generation based on a given natural language description. They have been integrated into open source projects and commercial products to facilitate daily coding activities. The natural language description in the prompt is crucial for LLMs to comprehend users\u2019 requirements. Prior studies have uncovered that LLMs are sensitive to changes in the prompts, including slight changes that look inconspicuous. However, the natural language descriptions often vary in real-world scenarios (e.g., different formats, grammar, and wording). Prior studies on the robustness of LLMs were often based on random perturbations, and such perturbations may not actually happen. In this article, we conduct a comprehensive study to investigate how code LLMs are robust to variations of natural language descriptions in real-world scenarios. We summarize 18 categories of perturbations of natural language and three combinations of co-occurred categories based on our literature review and online survey with practitioners. We propose an automated framework, NLPerturbator, which can perform perturbations of each category given a set of prompts. Through a series of experiments on code generation using sevencode LLMs, we find that the perturbed prompts can decrease the performance of code generation by a considerable margin. Our study highlights the importance of enhancing the robustness of LLMs to real-world variations in the prompts, as well as the essentiality of attentively constructing the prompts.<\/jats:p>","DOI":"10.1145\/3745764","type":"journal-article","created":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T09:25:42Z","timestamp":1751966742000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-9945-7729","authenticated-orcid":false,"given":"Junkai","family":"Chen","sequence":"first","affiliation":[{"name":"The State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China and Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4909-1535","authenticated-orcid":false,"given":"Zhenhao","family":"Li","sequence":"additional","affiliation":[{"name":"York University, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0093-3292","authenticated-orcid":false,"given":"Xing","family":"Hu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6302-3256","authenticated-orcid":false,"given":"Xin","family":"Xia","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,3,11]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00119"},{"key":"e_1_3_1_3_2","volume-title":"IEEE\/ACM 46th International Conference on Software Engineering (ICSE)","author":"Feng S.","year":"2024","unstructured":"S. Feng and C. Chen. 2024. Prompting is all your need: Automated android bug replay with large language models. In IEEE\/ACM 46th International Conference on Software Engineering (ICSE). IEEE."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608134"},{"key":"e_1_3_1_5_2","unstructured":"M. Chen J. Tworek H. Jun Q. Yuan H. P. de Oliveira Pinto J. Kaplan H. Edwards Y. Burda N. Joseph G. Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_1_6_2","unstructured":"R. Li L. B. Allal Y. Zi N. Muennighoff D. Kocetkov C. Mou M. Marone C. Akiki J. Li J. Chim et al. 2023. Starcoder: May the source be with you! arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_1_7_2","unstructured":"H. Touvron L. Martin K. Stone P. Albert A. Almahairi Y. Babaei N. Bashlykov S. Batra P. Bhargava S. Bhosale et\u00a0al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_1_8_2","unstructured":"F. Lin D. J. Kim and T.-H. (Peter) Chen. 2024. When LLM-based code generation meets the software development process. arXiv:2403.15852. Retrieved from https:\/\/arxiv.org\/abs\/2403.15852"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639085"},{"key":"e_1_3_1_10_2","unstructured":"X. Shen Z. Chen M. Backes and Y. Zhang. 2023. In ChatGPT we trust? measuring and characterizing the reliability of ChatGPT. arXiv:2304.08979. Retrieved from https:\/\/arxiv.org\/abs\/2304.08979"},{"key":"e_1_3_1_11_2","volume-title":"ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models","author":"Wang J.","year":"2023","unstructured":"J. Wang, H. Xixu, W. Hou, H. Chen, R. Zheng, Y. Wang, L. Yang, W. Ye, H. Huang, X. Geng, et al. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models."},{"key":"e_1_3_1_12_2","volume-title":"35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)","author":"Wang B.","unstructured":"B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)."},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"K. Zhu J. Wang J. Zhou Z. Wang H. Chen Y. Wang L. Yang W. Ye N. Z. Gong Y. Zhang et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv:2306.04528. Retrieved from https:\/\/arxiv.org\/abs\/2306.04528","DOI":"10.1145\/3689217.3690621"},{"key":"e_1_3_1_14_2","unstructured":"ChatGPT. 2023. OpenAI. Retrieved August 2023 from https:\/\/chat.openai.com"},{"key":"e_1_3_1_15_2","unstructured":"W. Ye M. Ou T. Li \u00a0Y. Chen X. Ma Y. Yanggong S. Wu J. Fu G. Chen H. Wang et al. 2023. Assessing hidden risks of LLMs: An empirical study on robustness consistency and credibility. arXiv:2305.10235. Retrieved from https:\/\/arxiv.org\/abs\/2305.10235"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.773"},{"key":"e_1_3_1_17_2","doi-asserted-by":"crossref","unstructured":"A. Mastropaolo L. Pascarella E. Guglielmi M. Ciniselli S. Scalabrino R. Oliveto and G. Bavota. 2023. On the robustness of code generation techniques: An empirical study on GitHub copilot. In 45th International Conference on Software Engineering.","DOI":"10.1109\/ICSE48619.2023.00181"},{"key":"e_1_3_1_18_2","unstructured":"L. Zhong and Z. Wang. 2023. A study on robustness and reliability of large language model code generation. arXiv:2308.10335. Retrieved from https:\/\/arxiv.org\/abs\/2308.10335"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501256"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.77"},{"key":"e_1_3_1_21_2","unstructured":"A. Shirafuji Y. Watanobe T. Ito M. Morishita Y. Nakamura Y. Oda and J. Suzuki. 2023. Exploring the robustness of large language models for solving programming problems. arXiv:2306.14583. Retrieved from https:\/\/arxiv.org\/abs\/2306.14583"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1057"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1041"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_1_26_2","first-page":"21314","article-title":"Coderl: Mastering code generation through pretrained models and deep reinforcement learning","volume":"35","author":"Le H.","year":"2022","unstructured":"H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 35, 21314\u201321328.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_27_2","unstructured":"E. Nijkamp H. Hayashi C. Xiong S. Savarese and Y. Zhou. 2023. Codegen2: Lessons for training LLMs on programming and natural languages. arXiv:2305.02309. Retrieved from https:\/\/arxiv.org\/abs\/2305.02309"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_1_29_2","unstructured":"B. Rozi\u00e8re J. Gehring F. Gloeckle S. Sootla I. Gat X. E. Tan Y. Adi J. Liu T. Remez J. Rapin et al. 2023. Code llama: Open foundation models for code. arXiv:2308.12950. Retrieved from https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6430"},{"key":"e_1_3_1_31_2","first-page":"311","volume-title":"40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni K.","year":"2002","unstructured":"K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In 40th Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_1_32_2","volume-title":"The 11th International Conference on Learning Representations","author":"Zhou S.","year":"2022","unstructured":"S. Zhou, U. Alon, F. F. Xu, Z. Jiang, and G. Neubig. 2022. Docprompting: Generating code by retrieving the docs. In The 11th International Conference on Learning Representations."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3675395"},{"key":"e_1_3_1_34_2","unstructured":"ArXiv e-Print Archive. 2023. ArXiv - Cornell University. Retrieved October 2023 from https:\/\/arxiv.org\/"},{"key":"e_1_3_1_35_2","unstructured":"J. Chen Z. Li X. Hu and X. Xia. 2025. Replication package. Retrieved April 2025 from https:\/\/github.com\/iCSawyer\/NLPerturbator"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00079"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00131"},{"key":"e_1_3_1_38_2","first-page":"361","volume-title":"35th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201920)","author":"Li Z.","year":"2020","unstructured":"Z. Li, T. Chen, and W. Shang. 2020. Where shall we log? Studying and suggesting logging locations in code blocks. In 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201920), 361\u2013372."},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.11613\/BM.2012.031"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_3"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510152"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00061"},{"key":"e_1_3_1_43_2","unstructured":"Microsoft forms. 2023. Microsoft. Retrieved February 2024 from https:\/\/forms.office.com\/"},{"key":"e_1_3_1_44_2","unstructured":"J. Austin A. Odena M. Nye M. Bosma H. Michalewski D. Dohan E. Jiang C. Cai M. Terry Q. Le et al. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_1_45_2","first-page":"140","volume-title":"2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE)","author":"Chen J.","year":"2024","unstructured":"J. Chen, Z. Pan, X. Hu, Z. Li, G. Li, and X. Xia. 2024. Reasoning runtime behavior of a program with LLM: How far are we? In 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 140\u2013152."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00133"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00136"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00078"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3060918"},{"key":"e_1_3_1_50_2","volume-title":"The 11th International Conference on Learning Representations","author":"Fried D.","year":"2022","unstructured":"D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. 2022. Incoder: A generative model for code infilling and synthesis. In The 11th International Conference on Learning Representations."},{"key":"e_1_3_1_51_2","volume-title":"The 12th International Conference on Learning Representations","author":"Luo Z.","unstructured":"Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The 12th International Conference on Learning Representations."},{"key":"e_1_3_1_52_2","unstructured":"Model \u2013 OpenAI api. 2025. OpenAI. Retrieved April 2025 from https:\/\/platform.openai.com\/docs\/models\/gpt-3.5-turbo"},{"key":"e_1_3_1_53_2","unstructured":"L. Ben Allal N. Muennighoff L. Kumar Umapathi B. Lipkin and L. von Werra. 2022. A framework for the evaluation of code generation models. Retrieved from https:\/\/github.com\/bigcode-project\/bigcode-evaluation-harness"},{"key":"e_1_3_1_54_2","volume-title":"International Conference on Learning Representations (ICLR)","author":"Belinkov Y.","year":"2018","unstructured":"Y. Belinkov and Y. Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661221"},{"key":"e_1_3_1_56_2","doi-asserted-by":"crossref","unstructured":"Y. Huang Z. Lin X. Liu Y. Gong S. Lu F. Lei Y. Liang Y. Shen C. Lin N. Duan et al. 2024. Competition-level problems are effective LLM evaluators. arXiv:2312.02143. Retrieved from https:\/\/arxiv.org\/abs\/2312.02143","DOI":"10.18653\/v1\/2024.findings-acl.803"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-emnlp.772"},{"key":"e_1_3_1_58_2","volume-title":"The 13th International Conference on Learning Representations","author":"Jain N.","unstructured":"N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The 13th International Conference on Learning Representations."},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00032"},{"key":"e_1_3_1_60_2","doi-asserted-by":"crossref","unstructured":"J.-B. D\u00f6derlein M. Acher D. E. Khelladi and B. Combemale. 2022. Piloting copilot and codex: Hot temperature cold prompts or black magic? arXiv:2210.14699. Retrieved from https:\/\/arxiv.org\/abs\/2210.14699","DOI":"10.2139\/ssrn.4496380"},{"key":"e_1_3_1_61_2","volume-title":"The 12th International Conference on Learning Representations","author":"Jimenez C. E.","year":"2024","unstructured":"C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. 2024. SWE-bench: Can language models resolve real-world GitHub issues? In The 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=VTF8yNQM66"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3745764","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T16:27:52Z","timestamp":1773246472000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3745764"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,11]]},"references-count":60,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,4,30]]}},"alternative-id":["10.1145\/3745764"],"URL":"https:\/\/doi.org\/10.1145\/3745764","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,11]]},"assertion":[{"value":"2024-08-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}