{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T15:14:54Z","timestamp":1784906094027,"version":"3.55.0"},"reference-count":72,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T00:00:00Z","timestamp":1753228800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T00:00:00Z","timestamp":1753228800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Politecnico di Bari"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Large Language Models have received a lot of attention in recent years due to their outstanding performance on various Natural Language Processing tasks. They can be used for lots of applications, including assistance in code generation tasks. Actual literature lacks an exhaustive analysis of the benefits and drawbacks of using a Large Language Model for the generation of simple and complex code. This paper aims to overcome the issue: we perform a Literature Review to explore the state-of-the-art of the proposed topic, answering 4 Research Questions. Using the PRISMA methodology, we reviewed 66 papers published between 2021 and 2023. Our analysis reveals Python\u2019s dominance as the preferred language and identifies a significant research gap in addressing ethical constraints. Additionally, we provide insights into the performance of models such as GPT-4 and CodeLlama, and their comparative utility in tasks ranging from debugging to multi-turn program synthesis. The findings offer a foundation for future research aimed at optimizing LLMs for code generation.<\/jats:p>","DOI":"10.1007\/s42979-025-04241-5","type":"journal-article","created":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T10:18:57Z","timestamp":1753265937000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Usage of Large Language Model for Code Generation Tasks: A Review"],"prefix":"10.1007","volume":"6","author":[{"given":"Stefano","family":"Bistarelli","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Fiore","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ivan","family":"Mercanti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marina","family":"Mongiello","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,7,23]]},"reference":[{"key":"4241_CR1","unstructured":"Abburi H, Suesserman M, Pudota N, et\u00a0al. Generative ai text classification using ensemble llm approaches. 2023. arXiv:2309.07755"},{"key":"4241_CR2","doi-asserted-by":"crossref","unstructured":"Adamopoulou E, Moussiades L. An overview of chatbot technology. In: IFIP international conference on artificial intelligence applications and innovations, Springer, 2020; pp 373\u2013383","DOI":"10.1007\/978-3-030-49186-4_31"},{"key":"4241_CR3","doi-asserted-by":"publisher","unstructured":"Ali H, Patel P, Obaitan I, et\u00a0al. Evaluating ChatGPT\u2019s Performance in Responding to Questions About Endoscopic Procedures for Patients. iGIE 2023 https:\/\/doi.org\/10.1016\/j.igie.2023.10.001","DOI":"10.1016\/j.igie.2023.10.001"},{"key":"4241_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.106304","volume":"123","author":"Z Alizadehsani","year":"2023","unstructured":"Alizadehsani Z, Ghaemi H, Shahraki A, et al. DCServCG: A data-centric service code generation using deep learning. Eng Appl Artif Intell. 2023;123: 106304. https:\/\/doi.org\/10.1016\/j.engappai.2023.106304.","journal-title":"Eng Appl Artif Intell"},{"key":"4241_CR5","unstructured":"Anand Y, Nussbaum Z, Duderstadt B, et\u00a0al. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. 2023 https:\/\/github.com\/nomic-ai\/gpt4all"},{"key":"4241_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2023.100161","volume":"5","author":"U Bezirhan","year":"2023","unstructured":"Bezirhan U, von Davier M. Automated reading passage generation with OpenAI\u2019s large language model. Computers and Education Artificial Intelligence. 2023;5: 100161. https:\/\/doi.org\/10.1016\/j.caeai.2023.100161.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR7","doi-asserted-by":"publisher","unstructured":"Bull C, Kharrufa A. Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it. IEEE Software 2023 pp 1\u20139. https:\/\/doi.org\/10.1109\/MS.2023.3300574","DOI":"10.1109\/MS.2023.3300574"},{"issue":"3","key":"4241_CR8","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1007\/s10270-023-01105-5","volume":"22","author":"J C\u00e1mara","year":"2023","unstructured":"C\u00e1mara J, Troya J, Burgue\u00f1o L, et al. On the assessment of generative AI in modeling tasks: An experience report with ChatGPT and UML. Softw Syst Model. 2023;22(3):781\u201393. https:\/\/doi.org\/10.1007\/s10270-023-01105-5.","journal-title":"Softw Syst Model"},{"key":"4241_CR9","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2023.121186","volume":"235","author":"L Caruccio","year":"2024","unstructured":"Caruccio L, Cirillo S, Polese G, et al. Can ChatGPT provide intelligent diagnoses? A comparative study between predictive models and ChatGPT to define a new medical diagnostic bot. Expert Syst Appl. 2024;235: 121186. https:\/\/doi.org\/10.1016\/j.eswa.2023.121186.","journal-title":"Expert Syst Appl"},{"key":"4241_CR10","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1016\/j.omtn.2023.06.019","volume":"33","author":"S Chatterjee","year":"2023","unstructured":"Chatterjee S, Bhattacharya M, Lee SS, et al. Can artificial intelligence-strengthened ChatGPT or other large language models transform nucleic acid research? Molecular Therapy - Nucleic Acids. 2023;33:205\u20137. https:\/\/doi.org\/10.1016\/j.omtn.2023.06.019.","journal-title":"Molecular Therapy - Nucleic Acids"},{"key":"4241_CR11","unstructured":"Chen W, Ma X, Wang X, et\u00a0al. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research 2023 https:\/\/openreview.net\/forum?id=YfZ4ZPt8zd"},{"key":"4241_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2022.100110","volume":"4","author":"C Diwan","year":"2023","unstructured":"Diwan C, Srinivasa S, Suri G, et al. AI-based learning content generation and learning pathway augmentation to increase learner engagement. Computers and Education Artificial Intelligence. 2023;4: 100110. https:\/\/doi.org\/10.1016\/j.caeai.2022.100110.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR13","doi-asserted-by":"crossref","unstructured":"Doughty J, Wan Z, Bompelli A, et\u00a0al. A comparative study of ai-generated (gpt-4) and human-crafted mcqs in programming education. arXiv preprint 2023. arXiv:2312.03173","DOI":"10.1145\/3636243.3636256"},{"key":"4241_CR14","volume":"71","author":"YK Dwivedi","year":"2023","unstructured":"Dwivedi YK, Kshetri N, Hughes L, et al. \u201cso what if chatgpt wrote it?\u2019\u2019 multidisciplinary perspectives on opportunities, challenges and implications of generative conversational ai for research, practice and policy. Int J Inf Manage. 2023;71: 102642.","journal-title":"Int J Inf Manage"},{"issue":"4","key":"4241_CR15","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1109\/MS.2023.3265877","volume":"40","author":"C Ebert","year":"2023","unstructured":"Ebert C, Louridas P. Generative AI for Software Practitioners. IEEE Softw. 2023;40(4):30\u20138. https:\/\/doi.org\/10.1109\/MS.2023.3265877.","journal-title":"IEEE Softw"},{"key":"4241_CR16","unstructured":"EDiGiacinto Localai repository. 2023 https:\/\/github.com\/go-skynet\/LocalAI"},{"key":"4241_CR17","doi-asserted-by":"publisher","DOI":"10.1016\/j.jrt.2023.100060","volume":"13","author":"DO Eke","year":"2023","unstructured":"Eke DO. ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology. 2023;13: 100060. https:\/\/doi.org\/10.1016\/j.jrt.2023.100060.","journal-title":"Journal of Responsible Technology"},{"issue":"4","key":"4241_CR18","doi-asserted-by":"publisher","first-page":"420","DOI":"10.5195\/jmla.2018.345","volume":"106","author":"MB Eriksen","year":"2018","unstructured":"Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (pico) as a search strategy tool on literature search quality: a systematic review. Journal of the Medical Library Association JMLA. 2018;106(4):420.","journal-title":"Journal of the Medical Library Association JMLA"},{"key":"4241_CR19","doi-asserted-by":"crossref","unstructured":"Espejel JL, Ettifouri EH, Alassan MSY, et\u00a0al. Gpt-3.5, gpt-4, or bard? evaluating llms reasoning ability in zero-shot setting and performance boosting through prompts. Natural Language Processing Journal 2023;5:100032","DOI":"10.1016\/j.nlp.2023.100032"},{"key":"4241_CR20","doi-asserted-by":"crossref","unstructured":"Fagbohun O, Harrison RM, Dereventsov A. An empirical categorization of prompting techniques for large language models: A practitioner\u2019s guide. arXiv preprint 2024 arXiv:2402.14837","DOI":"10.51219\/JAIMLD\/Oluwole-Fagbohun\/15"},{"key":"4241_CR21","doi-asserted-by":"crossref","unstructured":"Feng Y, Vanam S, Cherukupally M, et\u00a0al. Investigating code generation performance of chat-gpt with crowdsourcing social data. In: Proceedings of the 47th IEEE Computer Software and Applications Conference, 2023a;pp 1\u201310","DOI":"10.1109\/COMPSAC57700.2023.00117"},{"key":"4241_CR22","doi-asserted-by":"publisher","unstructured":"Feng Y, Vanam S, Cherukupally M, et\u00a0al. Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data. In: 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, Torino, Italy, 2023b; pp 876\u2013885, https:\/\/doi.org\/10.1109\/COMPSAC57700.2023.00117","DOI":"10.1109\/COMPSAC57700.2023.00117"},{"key":"4241_CR23","doi-asserted-by":"publisher","first-page":"1500","DOI":"10.1162\/tacl_a_00615","volume":"11","author":"NM Guerreiro","year":"2023","unstructured":"Guerreiro NM, Alves DM, Waldendorf J, et al. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics. 2023;11:1500\u201317.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"4241_CR24","doi-asserted-by":"crossref","unstructured":"Hadi MU, Qureshi R, Shah A, et\u00a0al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints 2023","DOI":"10.36227\/techrxiv.23589741.v1"},{"issue":"4","key":"4241_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/j.tbench.2023.100089","volume":"2","author":"A Haleem","year":"2022","unstructured":"Haleem A, Javaid M, Singh RP. An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil Transactions on Benchmarks Standards and Evaluations. 2022;2(4): 100089. https:\/\/doi.org\/10.1016\/j.tbench.2023.100089.","journal-title":"BenchCouncil Transactions on Benchmarks Standards and Evaluations"},{"key":"4241_CR26","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1016\/j.neucom.2021.01.043","volume":"438","author":"K Hong","year":"2021","unstructured":"Hong K, Uh Y, Byun H. ArrowGAN: Learning to generate videos by learning Arrow of Time. Neurocomputing. 2021;438:223\u201334. https:\/\/doi.org\/10.1016\/j.neucom.2021.01.043.","journal-title":"Neurocomputing"},{"issue":"1","key":"4241_CR27","doi-asserted-by":"publisher","DOI":"10.1016\/j.tbench.2023.100105","volume":"3","author":"M Javaid","year":"2023","unstructured":"Javaid M, Haleem A, Singh RP. ChatGPT for healthcare services: An emerging stage for an innovative perspective. BenchCouncil Transactions on Benchmarks Standards and Evaluations. 2023;3(1): 100105. https:\/\/doi.org\/10.1016\/j.tbench.2023.100105.","journal-title":"BenchCouncil Transactions on Benchmarks Standards and Evaluations"},{"key":"4241_CR28","doi-asserted-by":"publisher","unstructured":"Javaid M, Haleem A, Singh RP. A study on ChatGPT for Industry 4.0: Background, Potentials, Challenges, and Eventualities. Journal of Economy and Technology 2023b https:\/\/doi.org\/10.1016\/j.ject.2023.08.001","DOI":"10.1016\/j.ject.2023.08.001"},{"issue":"2","key":"4241_CR29","doi-asserted-by":"publisher","DOI":"10.1016\/j.tbench.2023.100115","volume":"3","author":"M Javaid","year":"2023","unstructured":"Javaid M, Haleem A, Singh RP, et al. Unlocking the opportunities through ChatGPT Tool towards ameliorating the education system. BenchCouncil Transactions on Benchmarks Standards and Evaluations. 2023;3(2): 100115. https:\/\/doi.org\/10.1016\/j.tbench.2023.100115.","journal-title":"BenchCouncil Transactions on Benchmarks Standards and Evaluations"},{"key":"4241_CR30","unstructured":"Jiang J, Wang F, Shen J, et\u00a0al. A survey on large language models for code generation. arXiv preprint 2024 arXiv:2406.00515"},{"key":"4241_CR31","unstructured":"jmorganca Ollama repository. 2023 https:\/\/github.com\/jmorganca\/ollama"},{"issue":"1","key":"4241_CR32","first-page":"214","volume":"614","author":"A Jo","year":"2023","unstructured":"Jo A. The promise and peril of generative ai. Nature. 2023;614(1):214\u20136.","journal-title":"Nature"},{"key":"4241_CR33","doi-asserted-by":"crossref","unstructured":"Katz DM, Bommarito MJ, Gao S, et\u00a0al. Gpt-4 passes the bar exam. Available at SSRN 4389233 2023","DOI":"10.2139\/ssrn.4389233"},{"issue":"3","key":"4241_CR34","doi-asserted-by":"publisher","DOI":"10.1016\/j.imr.2023.100977","volume":"12","author":"TH Kim","year":"2023","unstructured":"Kim TH, Kang JW, Lee MS. AI Chat bot - ChatGPT-4: A new opportunity and challenges in complementary and alternative medicine. Integrative Medicine Research. 2023;12(3): 100977. https:\/\/doi.org\/10.1016\/j.imr.2023.100977.","journal-title":"Integrative Medicine Research"},{"key":"4241_CR35","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2020.106553","volume":"114","author":"N K\u00f6bis","year":"2021","unstructured":"K\u00f6bis N, Mossink LD. Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry. Comput Hum Behav. 2021;114: 106553. https:\/\/doi.org\/10.1016\/j.chb.2020.106553.","journal-title":"Comput Hum Behav"},{"key":"4241_CR36","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2023.101861","volume":"99","author":"J Koco\u0144","year":"2023","unstructured":"Koco\u0144 J, Cichecki I, Kaszyca O, et al. ChatGPT: Jack of all trades, master of none. Information Fusion. 2023;99: 101861. https:\/\/doi.org\/10.1016\/j.inffus.2023.101861.","journal-title":"Information Fusion"},{"key":"4241_CR37","doi-asserted-by":"publisher","DOI":"10.1016\/j.dche.2023.100126","volume":"9","author":"ZY Kong","year":"2023","unstructured":"Kong ZY, Adi V, Segovia-Hern\u00e1ndez JG, et al. Complementary role of large language models in educating undergraduate design of distillation column: Methodology development. Digital Chemical Engineering. 2023;9: 100126. https:\/\/doi.org\/10.1016\/j.dche.2023.100126.","journal-title":"Digital Chemical Engineering"},{"key":"4241_CR38","doi-asserted-by":"crossref","unstructured":"Laban P, Kry\u015bci\u0144ski W, Agarwal D, et\u00a0al. Summedits: Measuring llm ability at factual reasoning through the lens of summarization. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023;pp 9662\u20139676","DOI":"10.18653\/v1\/2023.emnlp-main.600"},{"key":"4241_CR39","doi-asserted-by":"publisher","unstructured":"Lajk\u00f3 M, Csuvik V, Vid\u00e1cs L. Towards JavaScript program repair with generative pre-trained transformer (GPT-2). In: Proceedings of the Third International Workshop on Automated Program Repair. Association for Computing Machinery, New York, NY, USA, APR \u201922, 2022;pp 61\u201368, https:\/\/doi.org\/10.1145\/3524459.3527350","DOI":"10.1145\/3524459.3527350"},{"issue":"2","key":"4241_CR40","doi-asserted-by":"publisher","first-page":"172.e1","DOI":"10.1016\/j.ajog.2023.04.020","volume":"229","author":"SW Li","year":"2023","unstructured":"Li SW, Kemp MW, Logan S, et al. ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology. Am J Obstet Gynecol. 2023;229(2):172.e1-172.e12. https:\/\/doi.org\/10.1016\/j.ajog.2023.04.020.","journal-title":"Am J Obstet Gynecol"},{"key":"4241_CR41","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2023.100140","volume":"4","author":"Y Li","year":"2023","unstructured":"Li Y, Sha L, Yan L, et al. Can large language models write reflectively. Computers and Education Artificial Intelligence. 2023;4: 100140.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR42","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2023.100140","volume":"4","author":"Y Li","year":"2023","unstructured":"Li Y, Sha L, Yan L, et al. Can large language models write reflectively. Computers and Education Artificial Intelligence. 2023;4: 100140. https:\/\/doi.org\/10.1016\/j.caeai.2023.100140.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR43","doi-asserted-by":"publisher","unstructured":"Martins J, Branco F, Mamede H. Combining Low-Code Development with ChatGPT to Novel No-Code Approaches: A Focus-group Study. Intelligent Systems with Applications 2023;p 200289. https:\/\/doi.org\/10.1016\/j.iswa.2023.200289","DOI":"10.1016\/j.iswa.2023.200289"},{"key":"4241_CR44","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2023.103424","volume":"134","author":"T McIntosh","year":"2023","unstructured":"McIntosh T, Liu T, Susnjak T, et al. Harnessing GPT-4 for generation of cybersecurity GRC policies: A focus on ransomware attack mitigation. Computers & Security. 2023;134: 103424. https:\/\/doi.org\/10.1016\/j.cose.2023.103424.","journal-title":"Computers & Security"},{"key":"4241_CR45","doi-asserted-by":"publisher","unstructured":"Megahed FM, Chen YJ, Ferris JA, et\u00a0al. How generative AI models such as ChatGPT can be (mis)used in SPC practice, education, and research? An exploratory study. Quality Engineering 2023; pp 1\u201329. https:\/\/doi.org\/10.1080\/08982112.2023.2206479","DOI":"10.1080\/08982112.2023.2206479"},{"key":"4241_CR46","doi-asserted-by":"publisher","DOI":"10.1016\/j.pec.2023.107947","volume":"115","author":"J Menichetti","year":"2023","unstructured":"Menichetti J, Hillen MA, Papageorgiou A, et al. How can ChatGPT be used to support healthcare communication research? Patient Educ Couns. 2023;115: 107947. https:\/\/doi.org\/10.1016\/j.pec.2023.107947.","journal-title":"Patient Educ Couns"},{"key":"4241_CR47","unstructured":"Ni A, Iyer S, Radev D, et\u00a0al. LEVER: Learning to Verify Language-to-Code Generation with Execution. In: Proceedings of the 40th International Conference on Machine Learning. PMLR, 2023; pp 26106\u201326128"},{"key":"4241_CR48","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/j.neucom.2021.05.039","volume":"459","author":"LY Nie","year":"2021","unstructured":"Nie LY, Gao C, Zhong Z, et al. CoreGen: Contextualized Code Representation Learning for Commit Message Generation. Neurocomputing. 2021;459:97\u2013107. https:\/\/doi.org\/10.1016\/j.neucom.2021.05.039.","journal-title":"Neurocomputing"},{"key":"4241_CR49","unstructured":"Nijkamp E, Pang B, Hayashi H, et\u00a0al. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In: The Eleventh International Conference on Learning Representations 2022"},{"issue":"03","key":"4241_CR50","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/MS.2023.3248401","volume":"40","author":"I Ozkaya","year":"2023","unstructured":"Ozkaya I. Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications. IEEE Softw. 2023;40(03):4\u20138. https:\/\/doi.org\/10.1109\/MS.2023.3248401.","journal-title":"IEEE Softw"},{"issue":"04","key":"4241_CR51","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/MS.2023.3278056","volume":"40","author":"I Ozkaya","year":"2023","unstructured":"Ozkaya I. The Next Frontier in Software Development: AI-Augmented Software Development Processes. IEEE Softw. 2023;40(04):4\u20139. https:\/\/doi.org\/10.1109\/MS.2023.3278056.","journal-title":"IEEE Softw"},{"key":"4241_CR52","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmat.2023.08.007","author":"YJ Park","year":"2023","unstructured":"Park YJ, Kaplan D, Ren Z, et al. Can ChatGPT be used to generate scientific hypotheses? Journal of Materiomics. 2023. https:\/\/doi.org\/10.1016\/j.jmat.2023.08.007.","journal-title":"Journal of Materiomics"},{"key":"4241_CR53","unstructured":"Pothukuchi AS, Kota LV, Mallikarjunaradhya V. Impact of Generative AI on the Software Development Lifecycle (SDLC) 2023"},{"key":"4241_CR54","doi-asserted-by":"publisher","unstructured":"Sarkar A. Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models? In: Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. Association for Computing Machinery, New York, NY, USA, Onward! 2023, 2023;pp 153\u2013167, https:\/\/doi.org\/10.1145\/3622758.3622882","DOI":"10.1145\/3622758.3622882"},{"issue":"1","key":"4241_CR55","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13643-021-01671-z","volume":"10","author":"R Sarkis-Onofre","year":"2021","unstructured":"Sarkis-Onofre R, Catal\u00e1-L\u00f3pez F, Aromataris E, et al. How to properly use the prisma statement. Syst Rev. 2021;10(1):1\u20133.","journal-title":"Syst Rev"},{"key":"4241_CR56","doi-asserted-by":"publisher","unstructured":"Sarsa S, Denny P, Hellas A, et\u00a0al. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In: Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1, ICER \u201922, vol\u00a01. Association for Computing Machinery, New York, NY, USA, 2022;pp 27\u201343, https:\/\/doi.org\/10.1145\/3501385.3543957","DOI":"10.1145\/3501385.3543957"},{"key":"4241_CR57","doi-asserted-by":"publisher","DOI":"10.1016\/j.fsidi.2023.301609","volume":"46","author":"M Scanlon","year":"2023","unstructured":"Scanlon M, Breitinger F, Hargreaves C, et al. ChatGPT for digital forensic investigation: The good, the bad, and the unknown. Forensic Science International Digital Investigation. 2023;46: 301609. https:\/\/doi.org\/10.1016\/j.fsidi.2023.301609.","journal-title":"Forensic Science International Digital Investigation"},{"key":"4241_CR58","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1016\/j.jbusres.2021.05.009","volume":"134","author":"D Sj\u00f6din","year":"2021","unstructured":"Sj\u00f6din D, Parida V, Palmi\u00e9 M, et al. How AI capabilities enable business model innovation: Scaling AI through co-evolutionary processes and feedback loops. J Bus Res. 2021;134:574\u201387. https:\/\/doi.org\/10.1016\/j.jbusres.2021.05.009.","journal-title":"J Bus Res"},{"key":"4241_CR59","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2023.100160","volume":"5","author":"JL Steele","year":"2023","unstructured":"Steele JL. To GPT or not GPT? Empowering our students to learn with AI. Computers and Education Artificial Intelligence. 2023;5: 100160. https:\/\/doi.org\/10.1016\/j.caeai.2023.100160.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR60","doi-asserted-by":"publisher","unstructured":"Sun J, Liao QV, Muller M, et\u00a0al. Investigating Explainability of Generative AI for Code through Scenario-based Design. In: 27th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, IUI \u201922, 2022;pp 212\u2013228, https:\/\/doi.org\/10.1145\/3490099.3511119","DOI":"10.1145\/3490099.3511119"},{"key":"4241_CR61","doi-asserted-by":"publisher","unstructured":"Svyatkovskiy A, Deng SK, Fu S, et\u00a0al. IntelliCode compose: Code generation using transformer. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, ESEC\/FSE 2020, 2020;pp 1433\u20131443, https:\/\/doi.org\/10.1145\/3368089.3417058","DOI":"10.1145\/3368089.3417058"},{"key":"4241_CR62","unstructured":"symflower. Devqualityeval. 2024a https:\/\/github.com\/symflower\/eval-dev-quality"},{"key":"4241_CR63","unstructured":"symflower. Devqualityeval. 2024b. https:\/\/symflower.com\/en\/company\/blog\/2024\/dev-quality-eval-v0.6-o1-preview-is-the-king-of-code-generation-but-is-super-slow-and-expensive\/"},{"key":"4241_CR64","doi-asserted-by":"publisher","unstructured":"Taulli T. Auto Code Generation. In: Taulli T (ed) Generative AI: How ChatGPT and Other AI Tools Will Revolutionize Business. Apress, Berkeley, CA, 2023;p 127\u2013143, https:\/\/doi.org\/10.1007\/978-1-4842-9367-6_6","DOI":"10.1007\/978-1-4842-9367-6_6"},{"key":"4241_CR65","unstructured":"Topal MO, Bas A, van Heerden I. Exploring transformers in natural language generation: Gpt, bert, and xlnet. arXiv preprint 2021; arXiv:2102.08036"},{"key":"4241_CR66","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2022.100109","volume":"3","author":"AJ Vidanaralage","year":"2022","unstructured":"Vidanaralage AJ, Dharmaratne AT, Haque S. AI-based multidisciplinary framework to assess the impact of gamified video-based learning through schema and emotion analysis. Computers and Education Artificial Intelligence. 2022;3: 100109. https:\/\/doi.org\/10.1016\/j.caeai.2022.100109.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR67","doi-asserted-by":"publisher","unstructured":"Wang S, Lin B, Sun Z, et\u00a0al. Two Birds with One Stone: Boosting Code Generation and Code Search via a Generative Adversarial Network. Proceedings of the ACM on Programming Languages 7(OOPSLA2):239:486\u2013239:515. 2023 https:\/\/doi.org\/10.1145\/3622815","DOI":"10.1145\/3622815"},{"key":"4241_CR68","first-page":"24824","volume":"35","author":"J Wei","year":"2022","unstructured":"Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst. 2022;35:24824\u201337.","journal-title":"Adv Neural Inf Process Syst"},{"issue":"1","key":"4241_CR69","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/365153.365168","volume":"9","author":"J Weizenbaum","year":"1966","unstructured":"Weizenbaum J. Eliza-a computer program for the study of natural language communication between man and machine. Commun ACM. 1966;9(1):36\u201345.","journal-title":"Commun ACM"},{"key":"4241_CR70","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2023.100147","volume":"4","author":"R Yilmaz","year":"2023","unstructured":"Yilmaz R, Karaoglan Yilmaz FG. The effect of generative artificial intelligence (AI)-based tool use on students\u2019 computational thinking skills, programming self-efficacy and motivation. Computers and Education Artificial Intelligence. 2023;4: 100147. https:\/\/doi.org\/10.1016\/j.caeai.2023.100147.","journal-title":"Computers and Education Artificial Intelligence"},{"key":"4241_CR71","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2023.126708","volume":"557","author":"B Zhao","year":"2023","unstructured":"Zhao B, Jin W, Del Ser J, et al. ChatAgri: Exploring potentials of ChatGPT on cross-linguistic agricultural text classification. Neurocomputing. 2023;557: 126708. https:\/\/doi.org\/10.1016\/j.neucom.2023.126708.","journal-title":"Neurocomputing"},{"key":"4241_CR72","unstructured":"Zhuo TY. Large language models are state-of-the-art evaluators of code generation. arXiv preprint 2023. arXiv:2304.14317"}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-025-04241-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-025-04241-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-025-04241-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,7]],"date-time":"2025-09-07T19:41:55Z","timestamp":1757274115000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-025-04241-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,23]]},"references-count":72,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["4241"],"URL":"https:\/\/doi.org\/10.1007\/s42979-025-04241-5","relation":{},"ISSN":["2661-8907"],"issn-type":[{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,23]]},"assertion":[{"value":"1 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 July 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}}],"article-number":"673"}}