{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T15:32:06Z","timestamp":1776094326222,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":51,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,10,24]],"date-time":"2024-10-24T00:00:00Z","timestamp":1729728000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,10,24]]},"DOI":"10.1145\/3674805.3686689","type":"proceedings-article","created":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T18:39:24Z","timestamp":1729017564000},"page":"292-301","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Are Large Language Models a Threat to Programming Platforms? An Exploratory Study"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-0654-151X","authenticated-orcid":false,"given":"Md Mustakim","family":"Billah","sequence":"first","affiliation":[{"name":"University of Saskatchewan, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9470-4233","authenticated-orcid":false,"given":"Palash Ranjan","family":"Roy","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6715-3341","authenticated-orcid":false,"given":"Zadia","family":"Codabux","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1247-7781","authenticated-orcid":false,"given":"Banani","family":"Roy","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,10,24]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Chatgpt vs. bard: A comparative study. UMBC Student Collection","author":"Ahmed Imtiaz","year":"2023","unstructured":"Imtiaz Ahmed, Mashrafi Kajol, Uzma Hasan, Partha\u00a0Protim Datta, Ayon Roy, and Md\u00a0Rokonuzzaman Reza. 2023. Chatgpt vs. bard: A comparative study. UMBC Student Collection (2023)."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58846.2023.00013"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE43999.2019.9028518"},{"key":"e_1_3_2_1_4_1","volume-title":"GPT-4, Claude, and Bard. GPT-4, Claude, and Bard (June 12","author":"Borji Ali","year":"2023","unstructured":"Ali Borji and Mehrdad Mohammadian. 2023. Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and Bard. GPT-4, Claude, and Bard (June 12, 2023) (2023)."},{"key":"e_1_3_2_1_5_1","volume-title":"The goal question metric approach. Encyclopedia of software engineering","author":"Gianluigi Caldiera Victor R","year":"1994","unstructured":"Victor R Basili1\u00a0Gianluigi Caldiera and H\u00a0Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering (1994), 528\u2013532."},{"key":"e_1_3_2_1_6_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de\u00a0Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_3_2_1_7_1","volume-title":"28th International Conference on Evaluation and Assessment in Software Engineering(Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE\u201924)","author":"Coignion Tristan","year":"2024","unstructured":"Tristan Coignion, Cl\u00e9ment Quinton, and Romain Rouvoy. 2024. A Performance Study of LLM-Generated Code on C2. In EASE\u201924 - 28th International Conference on Evaluation and Assessment in Software Engineering(Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE\u201924)). Salerno, Italy. https:\/\/hal.science\/hal-04525620"},{"key":"e_1_3_2_1_8_1","volume-title":"Github copilot ai pair programmer: Asset or liability?Journal of Systems and Software 203","author":"Dakhel Arghavan\u00a0Moradi","year":"2023","unstructured":"Arghavan\u00a0Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel\u00a0C Desmarais, and Zhen Ming\u00a0Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability?Journal of Systems and Software 203 (2023), 111734."},{"key":"e_1_3_2_1_9_1","unstructured":"Hampus Ekedahl and Vilma Helander. 2023. Can artificial intelligence replace humans in programming?"},{"key":"e_1_3_2_1_10_1","volume-title":"Proceedings of the 47th IEEE Computer Software and Applications Conference. 1\u201310","author":"Feng Yunhe","year":"2023","unstructured":"Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of Chat-GPT with Crowdsourcing Social Data. In Proceedings of the 47th IEEE Computer Software and Applications Conference. 1\u201310."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511861.3511863"},{"key":"e_1_3_2_1_12_1","volume-title":"Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999","author":"Fried Daniel","year":"2022","unstructured":"Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022)."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","unstructured":"Felix HANS. [n. d.]. GPT-3.5\/4-Is the programming performance declining over time?focus 7 8 ([n. d.]) 9. https:\/\/doi.org\/10.13140\/RG.2.2.30950.80966","DOI":"10.13140\/RG.2.2.30950.80966"},{"key":"e_1_3_2_1_14_1","volume-title":"ChatGPT vs. Bard - Which is better at solving coding problems?10.13140\/RG.2.2.36200.65284 (08","author":"Hans Felix","year":"2023","unstructured":"Felix Hans. 2023. ChatGPT vs. Bard - Which is better at solving coding problems?10.13140\/RG.2.2.36200.65284 (08 2023)."},{"key":"e_1_3_2_1_15_1","volume-title":"A Software Engineer\u2019s Guide to Seniority: A Guide to Technical Leadership","author":"Harper Jocelyn","unstructured":"Jocelyn Harper. 2022. Interview insight: How to get the job. In A Software Engineer\u2019s Guide to Seniority: A Guide to Technical Leadership. Springer, 19\u201328."},{"key":"e_1_3_2_1_16_1","volume-title":"Whodunit: Classifying Code as Human Authored or GPT-4 Generated\u2013A case study on CodeChef problems. arXiv preprint arXiv:2403.04013","author":"Idialu Oseremen\u00a0Joy","year":"2024","unstructured":"Oseremen\u00a0Joy Idialu, Noble\u00a0Saji Mathews, Rungroj Maipradit, Joanne\u00a0M Atlee, and Mei Nagappan. 2024. Whodunit: Classifying Code as Human Authored or GPT-4 Generated\u2013A case study on CodeChef problems. arXiv preprint arXiv:2403.04013 (2024)."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.3390\/a17020062"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106633"},{"key":"e_1_3_2_1_19_1","volume-title":"Understanding the Relevance, Efficiency and Efficacy of Timed Coding Assessments in the Software Engineering Industry. Ph.\u00a0D. Dissertation","author":"Leon\u00a0Alarcon A","unstructured":"Paola\u00a0A Leon\u00a0Alarcon. 2024. Understanding the Relevance, Efficiency and Efficacy of Timed Coding Assessments in the Software Engineering Industry. Ph.\u00a0D. Dissertation. Massachusetts Institute of Technology."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWSC55060.2022.00010"},{"key":"e_1_3_2_1_21_1","volume-title":"Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161","author":"Li Raymond","year":"2023","unstructured":"Raymond Li, Loubna\u00a0Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161 (2023)."},{"key":"e_1_3_2_1_22_1","volume-title":"Competition-level code generation with alphacode. Science 378, 6624","author":"Li Yujia","year":"2022","unstructured":"Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\u00e9mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal\u00a0Lago, 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092\u20131097."},{"key":"e_1_3_2_1_23_1","volume-title":"Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36","author":"Liu Jiawei","year":"2024","unstructured":"Jiawei Liu, Chunqiu\u00a0Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3392499"},{"key":"e_1_3_2_1_25_1","volume-title":"Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568","author":"Luo Ziyang","year":"2023","unstructured":"Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 (2023)."},{"key":"e_1_3_2_1_26_1","volume-title":"Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36","author":"Madaan Aman","year":"2024","unstructured":"Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_2_1_27_1","volume-title":"The chi-square test of independence. Biochemia medica 23, 2","author":"McHugh L","year":"2013","unstructured":"Mary\u00a0L McHugh. 2013. The chi-square test of independence. Biochemia medica 23, 2 (2013), 143\u2013149."},{"key":"e_1_3_2_1_28_1","volume-title":"ICode-An Unified Competitive Coding Profile Platform. In 2024 International Conference on Emerging Smart Computing and Informatics (ESCI). IEEE, 1\u20135.","author":"Muke Saurav","year":"2024","unstructured":"Saurav Muke, Samruddhi Ahire, Geetanjali Kale, and Pranali Navghare. 2024. ICode-An Unified Competitive Coding Profile Platform. In 2024 International Conference on Emerging Smart Computing and Informatics (ESCI). IEEE, 1\u20135."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Nathalia Nascimento Paulo Alencar and Donald Cowan. 2023. Artificial Intelligence Versus Software Engineers: An Evidence-based Assessment Focusing on Non-functional Requirements. (2023).","DOI":"10.21203\/rs.3.rs-3126005\/v1"},{"key":"e_1_3_2_1_30_1","volume-title":"Comparing software developers with chatgpt: An empirical investigation. arXiv preprint arXiv:2305.11837","author":"Nascimento Nathalia","year":"2023","unstructured":"Nathalia Nascimento, Paulo Alencar, and Donald Cowan. 2023. Comparing software developers with chatgpt: An empirical investigation. arXiv preprint arXiv:2305.11837 (2023)."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528470"},{"key":"e_1_3_2_1_32_1","volume-title":"Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474","author":"Nijkamp Erik","year":"2022","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)."},{"key":"e_1_3_2_1_33_1","volume-title":"The End of an Era: Can Ai Subsume Software Developers. Evaluating Chatgpt and Copilot Capabilities Using Leetcode Problems","author":"Nikolaidis N","year":"2023","unstructured":"N Nikolaidis, K Flamos, D Feitosa, A Chatzigeorgiou, and A Ampatzoglou. 2023. The End of an Era: Can Ai Subsume Software Developers. Evaluating Chatgpt and Copilot Capabilities Using Leetcode Problems (2023)."},{"key":"e_1_3_2_1_34_1","volume-title":"LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828","author":"Ouyang Shuyin","year":"2023","unstructured":"Shuyin Ouyang, Jie\u00a0M Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828 (2023)."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.matpr.2021.05.596"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC_ASET53395.2022.9765916"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWSC60764.2023.00011"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICISET54810.2022.9775821"},{"key":"e_1_3_2_1_39_1","volume-title":"Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14","author":"Runeson Per","year":"2009","unstructured":"Per Runeson and Martin H\u00f6st. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14 (2009), 131\u2013164."},{"key":"e_1_3_2_1_40_1","volume-title":"Extending the frontier of chatgpt: Code generation and debugging. arXiv preprint arXiv:2307.08260","author":"Sakib Fardin\u00a0Ahsan","year":"2023","unstructured":"Fardin\u00a0Ahsan Sakib, Saadat\u00a0Hasan Khan, and AHM Karim. 2023. Extending the frontier of chatgpt: Code generation and debugging. arXiv preprint arXiv:2307.08260 (2023)."},{"key":"e_1_3_2_1_41_1","volume-title":"Chat GPT & Google Bard AI: A Review. In 2023 International Conference on IoT, Communication and Automation Technology (ICICAT). IEEE, 1\u20136.","author":"Singh Shashi\u00a0Kant","year":"2023","unstructured":"Shashi\u00a0Kant Singh, Shubham Kumar, and Pawan\u00a0Singh Mehra. 2023. Chat GPT & Google Bard AI: A Review. In 2023 International Conference on IoT, Communication and Automation Technology (ICICAT). IEEE, 1\u20136."},{"key":"e_1_3_2_1_42_1","volume-title":"Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT\u2013A Text-to-SQL Parsing Comparison. arXiv preprint arXiv:2310.10190","author":"Sun Shuo","year":"2023","unstructured":"Shuo Sun, Yuchen Zhang, Jiahuan Yan, Yuze Gao, Donovan Ong, Bin Chen, and Jian Su. 2023. Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT\u2013A Text-to-SQL Parsing Comparison. arXiv preprint arXiv:2310.10190 (2023)."},{"key":"e_1_3_2_1_43_1","volume-title":"Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805","author":"Team Gemini","year":"2023","unstructured":"Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew\u00a0M Dai, Anja Hauth, 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)."},{"key":"e_1_3_2_1_44_1","volume-title":"Errors to Boost for Reasoning. arXiv preprint arXiv:2403.20046","author":"Tong Yongqi","year":"2024","unstructured":"Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024. Can LLMs Learn from Previous Mistakes? Investigating LLMs\u2019 Errors to Boost for Reasoning. arXiv preprint arXiv:2403.20046 (2024)."},{"key":"e_1_3_2_1_45_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_2_1_47_1","volume-title":"Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922","author":"Wang Yue","year":"2023","unstructured":"Yue Wang, Hung Le, Akhilesh\u00a0Deepak Gotmare, Nghi\u00a0DQ Bui, Junnan Li, and Steven\u00a0CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023)."},{"key":"e_1_3_2_1_48_1","volume-title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven\u00a0CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3143560"},{"key":"e_1_3_2_1_50_1","volume-title":"Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc\u00a0V Le, Denny Zhou, 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824\u201324837."},{"key":"e_1_3_2_1_51_1","volume-title":"A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382","author":"White Jules","year":"2023","unstructured":"Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas\u00a0C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)."}],"event":{"name":"ESEM '24: ACM \/ IEEE International Symposium on Empirical Software Engineering and Measurement","location":"Barcelona Spain","acronym":"ESEM '24","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 18th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674805.3686689","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3674805.3686689","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T12:57:48Z","timestamp":1755867468000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674805.3686689"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,24]]},"references-count":51,"alternative-id":["10.1145\/3674805.3686689","10.1145\/3674805"],"URL":"https:\/\/doi.org\/10.1145\/3674805.3686689","relation":{},"subject":[],"published":{"date-parts":[[2024,10,24]]},"assertion":[{"value":"2024-10-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}