{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T16:40:00Z","timestamp":1783701600023,"version":"3.55.0"},"reference-count":132,"publisher":"Association for Computing Machinery (ACM)","issue":"8","funder":[{"name":"EU-funded project Sec4AI4Sec: Cybersecurity for AI-Augmented Systems","award":["101120393"],"award-info":[{"award-number":["101120393"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,11,30]]},"abstract":"<jats:p>\n            Large Language Models (LLMs) are gaining momentum in software development with prompt-driven programming enabling developers to create code from Natural Language (NL) instructions. However, studies have questioned their ability to produce secure code and, thereby, the quality of prompt-generated software. Alongside, various prompting techniques that carefully tailor prompts have emerged to elicit optimal responses from LLMs. Still, the interplay between such prompting strategies and secure code generation remains under-explored and calls for further investigations.\n            <jats:italic toggle=\"yes\">Objective<\/jats:italic>\n            : In this study, we investigate the impact of different prompting techniques on the security of code generated from NL instructions by LLMs.\n            <jats:italic toggle=\"yes\">Method<\/jats:italic>\n            : First, we perform a systematic literature review to identify the existing prompting techniques that can be used for code generation tasks. A subset of these techniques are evaluated on GPT-3, GPT-3.5, and GPT-4 models for secure code generation. For this, we used an existing dataset consisting of 150 NL security-relevant code generation prompts.\n            <jats:italic toggle=\"yes\">Results<\/jats:italic>\n            : Our work (i) classifies potential prompting techniques for code generation (ii) adapts and evaluates a subset of the identified techniques for secure code generation tasks, and (iii) observes a reduction in security weaknesses across the tested LLMs, especially after using an existing technique called Recursive Criticism and Improvement (RCI), contributing valuable insights to the ongoing discourse on LLM-generated code security.\n          <\/jats:p>","DOI":"10.1145\/3722108","type":"journal-article","created":{"date-parts":[[2025,3,7]],"date-time":"2025-03-07T15:57:21Z","timestamp":1741363041000},"page":"1-53","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["Prompting Techniques for Secure Code Generation: A Systematic Investigation"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9916-4456","authenticated-orcid":false,"given":"Catherine","family":"Tony","sequence":"first","affiliation":[{"name":"Hamburg University of Technology, Hamburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6304-771X","authenticated-orcid":false,"given":"Nicol\u00e1s E.","family":"D\u00edaz Ferreyra","sequence":"additional","affiliation":[{"name":"Hamburg University of Technology, Hamburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7633-0409","authenticated-orcid":false,"given":"Markus","family":"Mutas","sequence":"additional","affiliation":[{"name":"Hamburg University of Technology, Hamburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4157-8582","authenticated-orcid":false,"given":"Salem","family":"Dhif","sequence":"additional","affiliation":[{"name":"Hamburg University of Technology, Hamburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3591-7671","authenticated-orcid":false,"given":"Riccardo","family":"Scandariato","sequence":"additional","affiliation":[{"name":"Hamburg University of Technology, Hamburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,10,4]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"MITRE. 2024. MITRE. Retrieved October 10 2023 from https:\/\/www.mitre.org\/"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.NAACL-MAIN.211"},{"key":"e_1_3_2_4_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Arora Simran","year":"2023","unstructured":"Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher R\u00e9. 2023. Ask Me Anything: A simple strategy for prompting language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=bhUPJnS2g0X"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/S10664-023-10380-1"},{"key":"e_1_3_2_6_2","unstructured":"Jacob Austin Augustus Odena Maxwell I. Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le and Charles Sutton. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_7_2","unstructured":"Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell Jackson Kernion Andy Jones Anna Chen Anna Goldie Azalia Mirhoseini Cameron McKinnon et al. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. Retrieved from https:\/\/arxiv.org\/abs\/2212.08073"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.eacl-long.5"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D13-1160"},{"key":"e_1_3_2_11_2","unstructured":"Sumithra Bhakthavatsalam Daniel Khashabi Tushar Khot Bhavana Dalvi Mishra Kyle Richardson Ashish Sabharwal Carissa Schoenick Oyvind Tafjord and Peter Clark. 2021. Think you have solved direct-answer question answering? Try ARC-DA the direct-answer AI2 reasoning challenge. arXiv:2102.03315. Retrieved from https:\/\/arxiv.org\/abs\/2102.03315"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V34I05.6239"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Sid Black Stella Biderman Eric Hallahan Quentin Anthony Leo Gao Laurence Golding Horace He Connor Leahy Kyle McDonell Jason Phang et al. 2022. GPT-NeoX-20B: An open-source autoregressive language model. arXiv:2204.06745. Retrieved from https:\/\/arxiv.org\/abs\/2204.06745","DOI":"10.18653\/v1\/2022.bigscience-1.9"},{"key":"e_1_3_2_14_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920)","author":"Brown Tom B.","year":"2020","unstructured":"Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920), Virtual. Hugo Larochelle, Marc\u2019Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.)."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.mex.2022.101895"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Federico Cassano John Gouwar Daniel Nguyen Sydney Nguyen Luna Phipps-Costin Donald Pinckney Ming-Ho Yee Yangtian Zi Carolyn Jane Anderson Molly Q. Feldman et al. 2023. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49 7 (2023) 3675\u20133691. DOI: 10.1109\/TSE.2023.3267446","DOI":"10.1109\/TSE.2023.3267446"},{"key":"e_1_3_2_17_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D18-1241"},{"key":"e_1_3_2_19_2","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann et al. 2023. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research 24 (2023) 240:1\u2013240:113. Retrieved from http:\/\/jmlr.org\/papers\/v24\/22-1144.html"},{"key":"e_1_3_2_20_2","unstructured":"Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training verifiers to solve math word problems. arXiv:2110.14168. Retrieved from https:\/\/arxiv.org\/abs\/2110.14168"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2022.EMNLP-MAIN.222"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19-1423"},{"key":"e_1_3_2_23_2","unstructured":"Yihong Dong Jiazheng Ding Xue Jiang Zhuo Li Ge Li and Zhi Jin. 2023. CodeScore: Evaluating code generation by learning code execution. arXiv:2301.09043. Retrieved from https:\/\/arxiv.org\/abs\/2301.09043"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19-1246"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American Statistical Association 56 293 (1961) 52\u201364. Retrieved from http:\/\/www.jstor.org\/stable\/2282330","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.3115\/V1\/W14-3309"},{"key":"e_1_3_2_27_2","unstructured":"Ahmed Elnaggar Wei Ding Llion Jones Tom Gibbs Tamas Feher Christoph Angerer Silvia Severini Florian Matthes and Burkhard Rost. 2021. CodeTrans: Towards cracking the language of silicone\u2019s code through self-supervised deep learning and high performance computing. arXiv:2104.02443. Retrieved from https:\/\/arxiv.org\/abs\/2104.02443"},{"key":"e_1_3_2_28_2","unstructured":"Angela Fan Beliz Gokkaya Mark Harman Mitya Lyubarskiy Shubho Sengupta Shin Yoo and Jie M. Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv:2310.03533. Retrieved from https:\/\/arxiv.org\/abs\/2310.03533"},{"key":"e_1_3_2_29_2","first-page":"1536","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event","author":"Feng Zhangyin","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event. Trevor Cohn, Yulan He, and Yang Liu (Eds.), Association for Computational Linguistics, 1536\u20131547."},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Fried Daniel","year":"2023","unstructured":"Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A generative model for code infilling and synthesis. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=hQwb-lbM6EL"},{"key":"e_1_3_2_31_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Fu Yao","year":"2023","unstructured":"Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=yf1icZHC-l9"},{"key":"e_1_3_2_32_2","unstructured":"Deep Ganguli Amanda Askell Nicholas Schiefer Thomas I. Liao Kamile Lukosiute Anna Chen Anna Goldie Azalia Mirhoseini Catherine Olsson Danny Hernandez et al.. 2023. The capacity for moral self-correction in large language models. arXiv:2302.07459. Retrieved from https:\/\/arxiv.org\/abs\/2302.07459"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00370"},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the 9th International Conference on Learning Representations (ICLR \u201921)","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2021. GraphCodeBERT: Pre-training code representations with data flow. In Proceedings of the 9th International Conference on Learning Representations (ICLR \u201921), Virtual Event. OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=jLoC4ez43PZ"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.FINDINGS-ACL.55"},{"key":"e_1_3_2_36_2","unstructured":"Anne-Wil Harzing. 2016. Publish or Perish. Retrieved from https:\/\/harzing.com\/resources\/publish-or-perish"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2019.8870184"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3576915.3623175"},{"key":"e_1_3_2_39_2","unstructured":"Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song and Jacob Steinhardt. 2021. Measuring coding challenge competence with APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks\u201921) Virtual. Joaquin Vanschoren and Sai-Kit Yeung (Eds.). Retrieved from https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/hash\/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html"},{"key":"e_1_3_2_40_2","unstructured":"Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Dawn Song and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks \u201921) Virtual. Joaquin Vanschoren and Sai-Kit Yeung (Eds.). Retrieved from https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/hash\/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.3115\/V1\/D14-1058"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","unstructured":"Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023. Anna Rogers Jordan L. Boyd-Graber and Naoaki Okazaki (Eds.) Association for Computational Linguistics 1049\u20131065. DOI: 10.18653\/V1\/2023.FINDINGS-ACL.67","DOI":"10.18653\/V1\/2023.FINDINGS-ACL.67"},{"key":"e_1_3_2_43_2","first-page":"287","volume-title":"Proceedings of the Conference on Robot Learning (CoRL \u20192)","volume":"205","author":"Ichter Brian","year":"2022","unstructured":"Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, et al. 2022. Do as I can, not as I say: Grounding language in robotic affordances. In Proceedings of the Conference on Robot Learning (CoRL \u20192). Karen Liu, Dana Kulic, and Jeffrey Ichnowski (Eds.), Proceedings of Machine Learning Research, Vol. 205, PMLR, 287\u2013318. Retrieved from https:\/\/proceedings.mlr.press\/v205\/ichter23a.html"},{"key":"e_1_3_2_44_2","unstructured":"GitHub Inc. 2025. CodeQL. Retrieved January 9 2024 from https:\/\/codeql.github.com\/docs\/"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510203"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.EMNLP-MAIN.482"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00082"},{"key":"e_1_3_2_48_2","unstructured":"Xue Jiang Yihong Dong Lecheng Wang Zheng Fang Qiwei Shang Ge Li Zhi Jin and Wenpin Jiao. 2023. Self-planning code generation with large language models. arXiv:2303.06689. Retrieved from https:\/\/arxiv.org\/abs\/2303.06689"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P17-1147"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.FINDINGS-EMNLP.171"},{"key":"e_1_3_2_51_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923","author":"Kim Geunwoo","year":"2023","unstructured":"Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. In Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923). Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/7cc1005ec73cfbaac9fa21192b622507-Abstract-Conference.html"},{"key":"e_1_3_2_52_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS \u201922)","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS \u201922). Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00160"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","unstructured":"William H. Kruskal and W. Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association 47 260 (1952) 583\u2013621. DOI: 10.1080\/01621459.1952.10483441","DOI":"10.1080\/01621459.1952.10483441"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00276"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D17-1082"},{"key":"e_1_3_2_57_2","first-page":"2879","volume-title":"Proceedings of the 35th International Conference on Machine Learning (ICML \u201918)","volume":"80","author":"Lake Brenden M.","year":"2018","unstructured":"Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning (ICML \u201918). Jennifer G. Dy and Andreas Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, PMLR, 2879\u20132888. Retrieved from http:\/\/proceedings.mlr.press\/v80\/lake18a.html"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2022.FINDINGS-EMNLP.38"},{"key":"e_1_3_2_59_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS\u00a0\u201922)","author":"Le Hung","year":"2022","unstructured":"Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS\u00a0\u201922). Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/8636419dea1aa9fbd25fc4248e702da4-Abstract-Conference.html"},{"key":"e_1_3_2_60_2","volume-title":"Natural Language Processing with Transformers","author":"von Werra Lewis Tunstall Leandro","year":"2022","unstructured":"Leandro von Werra Lewis Tunstall and Thomas Wolf. 2022. Natural Language Processing with Transformers. O\u2019Reilly Media, Inc. (2022)."},{"key":"e_1_3_2_61_2","unstructured":"Opher Lieber Or Sharir Barak Lenz and Yoav Shoham. 2021. Jurassic-1: Technical Details and Evaluation. AI21 Labs Technical Report. Retrieved from https:\/\/uploads-ssl.webflow.com\/60fd4503684b466578c0d307\/61138924626a6981ee09caf6_jurassic_tech_paper.pdf"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.FINDINGS-EMNLP.165"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P17-1015"},{"key":"e_1_3_2_64_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923)","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923). Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html"},{"key":"e_1_3_2_65_2","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin B. Clement Dawn Drain Daxin Jiang Duyu Tang et al. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks \u201921) Virtual. Joaquin Vanschoren and Sai-Kit Yeung (Eds.). Retrieved from https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/hash\/c16a5320fa475530d9583c34fd356ef5-Abstract-round1.html"},{"key":"e_1_3_2_66_2","unstructured":"Aman Madaan Alexander Shypula Uri Alon Milad Hashemi Parthasarathy Ranganathan Yiming Yang Graham Neubig and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv:2302.07867. Retrieved from https:\/\/arxiv.org\/abs\/2302.07867"},{"key":"e_1_3_2_67_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923","author":"Madaan Aman","year":"2023","unstructured":"Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. In Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923). Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1111\/JCAL.12662"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.3115\/1075812.1075835"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.sigdial-1.28"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D18-1260"},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","unstructured":"Nasrin Mostafazadeh Nathanael Chambers Xiaodong He Devi Parikh Dhruv Batra Lucy Vanderwende Pushmeet Kohli and James F. Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv:1604.01696. Retrieved from http:\/\/arxiv.org\/abs\/1604.01696","DOI":"10.18653\/v1\/N16-1098"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.ACL-MAIN.441"},{"key":"e_1_3_2_74_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P16-1144"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.NAACL-MAIN.168"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46214.2022.9833571"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46215.2023.00001"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3576915.3623157"},{"key":"e_1_3_2_80_2","doi-asserted-by":"crossref","unstructured":"Long N. Phan Hieu Tran Daniel Le Hieu Nguyen James T. Anibal Alec Peltekian and Yanfang Ye. 2021. CoTexT: Multi-task learning with code-text transformer. arXiv:2105.08645. Retrieved from https:\/\/arxiv.org\/abs\/2105.08645","DOI":"10.18653\/v1\/2021.nlp4prog-1.5"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.EACL-MAIN.277"},{"key":"e_1_3_2_82_2","unstructured":"Ruchir Puri David S. Kung Geert Janssen Wei Zhang Giacomo Domeniconi Vladimir Zolotov Julian Dolby Jie Chen Mihir R. Choudhury Lindsey Decker et al. 2021. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks \u201921) Virtual. Joaquin Vanschoren and Sai-Kit Yeung (Eds.). Retrieved from https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/hash\/a5bfc9e07964f8dddeb95fc584cd965d-Abstract-round2.html"},{"key":"e_1_3_2_83_2","unstructured":"PyCQA. 2025. Bandit Documentation. Retrieved January 5 2024 from https:\/\/bandit.readthedocs.io\/en\/latest\/index.html"},{"key":"e_1_3_2_84_2","unstructured":"Alec Radford Jeff Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:160025533"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2019.00087"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P18-2124"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","unstructured":"Irum Rauf Marian Petre Thein Tun Tamara Lopez Paul Lunn Dirk van der Linden John N. Towse Helen Sharp Mark Levine Awais Rashid et al. 2022. The case for adaptive security interventions. ACM Transactions on Software Engineering and Methodology 31 1 (2022) 9:1\u20139:52. DOI: 10.1145\/3471930","DOI":"10.1145\/3471930"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00266"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3451760"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2024.ACL-LONG.761"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D15-1202"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/PST52912.2021.9647791"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474381"},{"key":"e_1_3_2_94_2","first-page":"2205","volume-title":"Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923)","author":"Sandoval Gustavo","year":"2023","unstructured":"Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. 2023. Lost at C: A user study on the security implications of large language model code assistants. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security \u201923). Joseph A. Calandrino and Carmela Troncoso (Eds.), USENIX Association, 2205\u20132222. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity23\/presentation\/sandoval"},{"key":"e_1_3_2_95_2","first-page":"127","volume-title":"Proceedings of the 33rd Annual Workshop of the Psychology of Programming Interest Group (PPIG \u201922)","author":"Sarkar Advait","year":"2022","unstructured":"Advait Sarkar, Carina Negreanu, Ben Zorn, Sruti Srinivasa Ragavan, Christian P\u00f6litz, and Andrew D. Gordon. 2022. What is it like to program with artificial intelligence? In Proceedings of the 33rd Annual Workshop of the Psychology of Programming Interest Group (PPIG \u201922). Simon Holland, Marian Petre, Luke Church, and Mariana Marasoiu (Eds.), Psychology of Programming Interest Group, 127\u2013153. Retrieved from https:\/\/ppig.org\/papers\/2022-ppig-33rd-sarkar\/"},{"key":"e_1_3_2_96_2","unstructured":"William Saunders Catherine Yeh Jeff Wu Steven Bills Long Ouyang Jonathan Ward and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv:2206.05802. Retrieved from https:\/\/arxiv.org\/abs\/2206.05802"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643991.3645071"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3549035.3561184"},{"key":"e_1_3_2_99_2","unstructured":"Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid Adam Fisch Adam R. Brown Adam Santoro Aditya Gupta Adri\u00e0 Garriga-Alonso et al. 2022. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. Retrieved from https:\/\/arxiv.org\/abs\/2206.04615"},{"key":"e_1_3_2_100_2","first-page":"20841","volume-title":"Proceedings of the International Conference on Machine Learning (ICML \u201922)","volume":"162","author":"Sun Tianxiang","year":"2022","unstructured":"Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for language-model-as-a-service. In Proceedings of the International Conference on Machine Learning (ICML \u201922). Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesv\u00e1ri, Gang Niu, and Sivan Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, PMLR, 20841\u201320855. Retrieved from https:\/\/proceedings.mlr.press\/v162\/sun22e.html"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.FINDINGS-ACL.824"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19-1421"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-8-45"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS57517.2022.00094"},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00084"},{"key":"e_1_3_2_106_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. LLaMA: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_2_107_2","unstructured":"Peter D. Turney Michael L. Littman Jeffrey Bigham and Victor Shnayder. 2003. Combining independent modules to solve multiple-choice synonym and analogy problems. arXiv:cs\/0309035. Retrieved from https:\/\/arxiv.org\/abs\/cs\/0309035"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_2_109_2","first-page":"3261","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS \u201919)","author":"Wang Alex","year":"2019","unstructured":"Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS \u201919). Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d\u2019Alch\u00e9-Buc, Emily B. Fox, and Roman Garnett (Eds.), 3261\u20133275. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/4496bf24afe7fab6f046bf4923da8de6-Abstract.html"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.174"},{"key":"e_1_3_2_111_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Wang Xuezhi","year":"2023","unstructured":"Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=1PL1NIMMrw"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_113_2","unstructured":"Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter Fei Xia Ed H. Chi Quoc V. Le and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS \u201922). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html"},{"key":"e_1_3_2_114_2","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv:2302.11382. Retrieved from https:\/\/arxiv.org\/abs\/2302.11382"},{"key":"e_1_3_2_115_2","doi-asserted-by":"crossref","unstructured":"Jules White Sam Hays Quchen Fu Jesse Spencer-Smith and Douglas C. Schmidt. 2023. ChatGPT prompt patterns for improving code quality refactoring requirements elicitation and software design. arXiv:2303.07839. Retrieved from https:\/\/arxiv.org\/abs\/2303.07839","DOI":"10.1007\/978-3-031-55642-5_4"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1145\/3475716.3484195"},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00023"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.1177\/0739456X17723971"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2022.EMNLP-MAIN.559"},{"key":"e_1_3_2_122_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS \u201923). Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html"},{"key":"e_1_3_2_123_2","unstructured":"Yifan Yao Jinhao Duan Kaidi Xu Yuanfang Cai Eric Sun and Yue Zhang. 2023. A survey on Large Language Model (LLM) security and privacy: The good the bad and the ugly. arXiv:2312.02003. Retrieved from https:\/\/arxiv.org\/abs\/2312.02003"},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00163"},{"key":"e_1_3_2_125_2","unstructured":"Burak Yeti\u015ftiren I\u015f\u0131k \u00d6zsoy Miray Ayerdem and Eray T\u00fcz\u00fcn. 2023. Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot Amazon CodeWhisperer and ChatGPT. arXiv:2304.10778. Retrieved from https:\/\/arxiv.org\/abs\/2304.10778"},{"key":"e_1_3_2_126_2","first-page":"40373","volume-title":"Proceedings of the International Conference on Machine Learning (ICML \u201923)","volume":"202","author":"Yu Zhiyuan","year":"2023","unstructured":"Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yevgeniy Vorobeychik, and Chaowei Xiao. 2023. CodeIPPrompt: Intellectual property infringement assessment of code language models. In Proceedings of the International Conference on Machine Learning (ICML \u201923). Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, PMLR, 40373\u201340389. Retrieved from https:\/\/proceedings.mlr.press\/v202\/yu23g.html"},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P19-1472"},{"key":"e_1_3_2_128_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534390"},{"key":"e_1_3_2_129_2","unstructured":"Xiang Zhang Junbo Jake Zhao and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015. Corinna Cortes Neil D. Lawrence Daniel D. Lee Masashi Sugiyama and Roman Garnett (Eds.) 649\u2013657. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2015\/hash\/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html"},{"key":"e_1_3_2_130_2","unstructured":"Chuanyang Zheng Zhengying Liu Enze Xie Zhenguo Li and Yu Li. 2023. Progressive-hint prompting improves reasoning in large language models. arXiv:2304.09797. Retrieved from https:\/\/arxiv.org\/abs\/2304.09797"},{"key":"e_1_3_2_131_2","doi-asserted-by":"crossref","unstructured":"Qinkai Zheng Xiao Xia Xu Zou Yuxiao Dong Shan Wang Yufei Xue Zihan Wang Lei Shen Andi Wang Yang Li et al. 2023. CodeGeeX: A pre-trained model for code generation with multilingual evaluations on HumanEval-X. arXiv:2303.17568. Retrieved from https:\/\/arxiv.org\/abs\/2303.17568","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_2_132_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Zhou Denny","year":"2023","unstructured":"Denny Zhou, Nathanael Sch\u00e4rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, et al. 2023. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=WZH7099tgfM"},{"key":"e_1_3_2_133_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)","author":"Zhou Yongchao","year":"2023","unstructured":"Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In Proceedings of the 11th International Conference on Learning Representations (ICLR \u201923)). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=92gvk82DE-"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3722108","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,4]],"date-time":"2025-10-04T11:08:12Z","timestamp":1759576092000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3722108"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,4]]},"references-count":132,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2025,11,30]]}},"alternative-id":["10.1145\/3722108"],"URL":"https:\/\/doi.org\/10.1145\/3722108","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,4]]},"assertion":[{"value":"2024-05-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}