{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T07:17:38Z","timestamp":1783408658661,"version":"3.54.6"},"publisher-location":"New York, NY, USA","reference-count":37,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T00:00:00Z","timestamp":1775952000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Accelerate Program for Scientific Discovery","award":["n\/a"],"award-info":[{"award-number":["n\/a"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,12]]},"DOI":"10.1145\/3786181.3788724","type":"proceedings-article","created":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T06:28:24Z","timestamp":1783405704000},"page":"59-66","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Code Roulette: How Prompt Variability Affects LLM Code Generation"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3703-8163","authenticated-orcid":false,"given":"Andrei","family":"Paleyes","sequence":"first","affiliation":[{"name":"Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7468-0123","authenticated-orcid":false,"given":"Diana","family":"Robinson","sequence":"additional","affiliation":[{"name":"Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4072-2828","authenticated-orcid":false,"given":"Radzim","family":"Sendyka","sequence":"additional","affiliation":[{"name":"Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6954-6859","authenticated-orcid":false,"given":"Christian","family":"Cabrera","sequence":"additional","affiliation":[{"name":"Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9258-1030","authenticated-orcid":false,"given":"Neil D.","family":"Lawrence","sequence":"additional","affiliation":[{"name":"Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,6]]},"reference":[{"key":"e_1_3_3_2_2_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et\u00a0al. 2021. Program synthesis with large language models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2108.07732 (2021)."},{"key":"e_1_3_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Pak Yuen\u00a0Patrick Chan Jacky Keung and Zhen Yang. 2025. Effectiveness of symmetric metamorphic relations on validating the stability of code generation LLM. Journal of Systems and Software 222 (2025) 112330.","DOI":"10.1016\/j.jss.2024.112330"},{"key":"e_1_3_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Junkai Chen Li Zhenhao Hu Xing and Xia Xin. 2025. NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations. ACM Trans. Softw. Eng. Methodol. (July 2025). 10.1145\/3745764Just Accepted.","DOI":"10.1145\/3745764"},{"key":"e_1_3_3_2_5_2","unstructured":"Liguo Chen Qi Guo Hongrui Jia Zhengran Zeng Xin Wang Yijiang Xu Jian Wu Yidong Wang Qing Gao Jindong Wang et\u00a0al. 2024. A Survey on Evaluating Large Language Models in Code Generation Tasks. arXiv e-prints (2024) arXiv\u20132408."},{"key":"e_1_3_3_2_6_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique\u00a0Ponde de Oliveira\u00a0Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe\u00a0Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William\u00a0Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew\u00a0N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (2021). arxiv:https:\/\/arXiv.org\/abs\/2107.03374\u00a0[cs.LG]"},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Mingzhe Du Anh\u00a0Tuan Luu Bin Ji Xiaobao Wu Dong Huang Terry\u00a0Yue Zhuo Qian Liu and See-Kiong Ng. 2025. CodeArena: A Collective Evaluation Platform for LLM Code Generation. CoRR abs\/2503.01295 (March 2025). 10.48550\/arXiv.2503.01295","DOI":"10.48550\/arXiv.2503.01295"},{"key":"e_1_3_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Ramtin Ehsani Sakshi Pathak and Preetha Chatterjee. 2025. Towards detecting prompt knowledge gaps for improved LLM-guided issue resolution. arXiv [cs.SE] (Jan. 2025).","DOI":"10.1109\/MSR66628.2025.00107"},{"key":"e_1_3_3_2_9_2","doi-asserted-by":"crossref","unstructured":"Mikhail Evtikhiev Egor Bogomolov Yaroslav Sokolov and Timofey Bryksin. 2023. Out of the Bleu: how should we assess quality of the code generation models?Journal of Systems and Software 203 (2023) 111741.","DOI":"10.1016\/j.jss.2023.111741"},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"crossref","unstructured":"Milton Friedman. 1940. A comparison of alternative tests of significance for the problem of m rankings. The annals of mathematical statistics 11 1 (1940) 86\u201392.","DOI":"10.1214\/aoms\/1177731944"},{"key":"e_1_3_3_2_11_2","volume-title":"The 4th Workshop on Mathematical Reasoning and AI at NeurIPS\u201924","author":"Gulati Aryan","year":"2024","unstructured":"Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes\u00a0Dumont, and Sanmi Koyejo. 2024. Putnam-AXIOM: A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS\u201924."},{"key":"e_1_3_3_2_12_2","doi-asserted-by":"crossref","unstructured":"Ava Heinonen Bettina Lehtel\u00e4 Arto Hellas and Fabian Fagerholm. 2023. Synthesizing research on programmers\u2019 mental models of programs tasks and concepts \u2014 A systematic literature review. Inf. Softw. Technol. 164 107300 (Dec. 2023) 107300.","DOI":"10.1016\/j.infsof.2023.107300"},{"key":"e_1_3_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671932"},{"key":"e_1_3_3_2_14_2","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Jain Naman","year":"2025","unstructured":"Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=chfJJYC3iL"},{"key":"e_1_3_3_2_15_2","volume-title":"The Twelfth International Conference on Learning Representations","author":"Jimenez Carlos\u00a0E","year":"2023","unstructured":"Carlos\u00a0E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik\u00a0R Narasimhan. 2023. SWE-bench: Can Language Models Resolve Real-world GitHub Issues?. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Ranim Khojah Francisco\u00a0Gomes de Oliveira\u00a0Neto Mazen Mohamad and Philipp Leitner. 2025. The Impact of Prompt Programming on Function-Level Code Generation. IEEE Transactions on Software Engineering 51 8 (2025) 2381\u20132395. 10.1109\/TSE.2025.3587794","DOI":"10.1109\/TSE.2025.3587794"},{"key":"e_1_3_3_2_17_2","doi-asserted-by":"crossref","unstructured":"William\u00a0H Kruskal and W\u00a0Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association 47 260 (1952) 583\u2013621.","DOI":"10.1080\/01621459.1952.10483441"},{"key":"e_1_3_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-1837"},{"key":"e_1_3_3_2_19_2","unstructured":"Jenny\u00a0T Liang Melissa Lin Nikitha Rao and Brad\u00a0A Myers. 2024. Prompts are programs too! Understanding how developers build software containing prompts. arXiv [cs.SE] (Sept. 2024)."},{"key":"e_1_3_3_2_20_2","unstructured":"Edward Ma. 2019. NLP Augmentation. https:\/\/github.com\/makcedward\/nlpaug."},{"key":"e_1_3_3_2_21_2","doi-asserted-by":"crossref","unstructured":"George\u00a0A Miller. 1995. WordNet: a lexical database for English. Commun. ACM 38 11 (1995) 39\u201341.","DOI":"10.1145\/219717.219748"},{"key":"e_1_3_3_2_22_2","unstructured":"Lincoln Murr Morgan Grainger and David Gao. 2023. Testing LLMS on code generation with varying levels of prompt specificity. URL https:\/\/arxiv.org\/abs\/2311.07599 (2023)."},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"crossref","unstructured":"Maureen\u00a0D Neumann Lisa Dion and Robert Snapp. 2021. Teaching Computational Thinking. https:\/\/mitpress.mit.edu\/9780262045056\/teaching-computational-thinking\/. Accessed: 2025-5-15.","DOI":"10.7551\/mitpress\/11209.001.0001"},{"key":"e_1_3_3_2_24_2","volume-title":"Psychometric Theory (3rd ed.)","author":"Nunnally Jum\u00a0C.","year":"1994","unstructured":"Jum\u00a0C. Nunnally and Ira\u00a0H. Bernstein. 1994. Psychometric Theory (3rd ed.). McGraw-Hill, New York. xxiv + 752 pages."},{"key":"e_1_3_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649165.3690111"},{"key":"e_1_3_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6319"},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-88714-7_29"},{"key":"e_1_3_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Diana Robinson Christian Cabrera Andrew\u00a0D Gordon Neil\u00a0D Lawrence and Lars Mennen. 2025. Requirements are all you need: The final frontier for end-user software engineering. ACM Transactions on Software Engineering and Methodology 34 5 (2025) 1\u201322.","DOI":"10.1145\/3708524"},{"key":"e_1_3_3_2_29_2","volume-title":"The Twelfth International Conference on Learning Representations","author":"Sclar Melanie","year":"2023","unstructured":"Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying Language Models\u2019 Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_3_2_30_2","unstructured":"Radzim Sendyka Christian Cabrera Andrei Paleyes Diana Robinson and Neil Lawrence. 2025. LLM Performance for Code Generation on Noisy Tasks. arxiv:https:\/\/arXiv.org\/abs\/2505.23598\u00a0[cs.LG] https:\/\/arxiv.org\/abs\/2505.23598"},{"key":"e_1_3_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-short.3"},{"key":"e_1_3_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Hari Subramonyam Divy Thakkar Andrew Ku J\u00fcrgen Dieber and Anoop Sinha. 2024. Prototyping with prompts: Emerging approaches and challenges in generative AI design for collaborative software teams. arXiv [cs.HC] (Feb. 2024).","DOI":"10.1145\/3706598.3713166"},{"key":"e_1_3_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653584"},{"key":"e_1_3_3_2_34_2","unstructured":"Xinchen Wang Pengfei Gao Chao Peng Ruida Hu and Cuiyun Gao. 2025. CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation. arXiv e-prints (2025) arXiv\u20132504."},{"key":"e_1_3_3_2_35_2","doi-asserted-by":"crossref","unstructured":"Jeannette\u00a0M Wing. 2008. Computational thinking and thinking about computing. Philos. Trans. A Math. Phys. Eng. Sci. 366 1881 (Oct. 2008) 3717\u20133725.","DOI":"10.1098\/rsta.2008.0118"},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"publisher","unstructured":"Sangyeop Yeo Yu-Seung Ma Sang\u00a0Cheol Kim Hyungkook Jun and Taeho Kim. 2024. Framework for evaluating code generation ability of large language models. ETRI Journal 46 1 (2024) 106\u2013117. 10.4218\/etrij.2023-0357","DOI":"10.4218\/etrij.2023-0357"},{"key":"e_1_3_3_2_37_2","volume-title":"International Conference on Learning Representations","author":"Zhang Tianyi","year":"2020","unstructured":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian\u00a0Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations."},{"key":"e_1_3_3_2_38_2","unstructured":"Ziyin Zhang Chaoyu Chen Bingchang Liu Cong Liao Zi Gong Hang Yu Jianguo Li and Rui Wang. 2023. Unifying the perspectives of nlp and software engineering: A survey on language models for code. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2311.07989 (2023)."}],"event":{"name":"LLM4Code '26: 3rd International Workshop on Large Language Models For Code","location":"Rio de Janeiro Brazil","acronym":"LLM4Code '26","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS","Faculty of Engineering of University of Porto"]},"container-title":["Proceedings of the 3rd International Workshop on Large Language Models For Code"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786181.3788724","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T06:41:03Z","timestamp":1783406463000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786181.3788724"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,12]]},"references-count":37,"alternative-id":["10.1145\/3786181.3788724","10.1145\/3786181"],"URL":"https:\/\/doi.org\/10.1145\/3786181.3788724","relation":{},"subject":[],"published":{"date-parts":[[2026,4,12]]},"assertion":[{"value":"2026-07-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}