{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T01:39:35Z","timestamp":1785548375408,"version":"3.56.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,11,30]],"date-time":"2024-11-30T00:00:00Z","timestamp":1732924800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2024,11,30]],"date-time":"2024-11-30T00:00:00Z","timestamp":1732924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["ScaDS.AI"],"award-info":[{"award-number":["ScaDS.AI"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["SI 2171\/2-2"],"award-info":[{"award-number":["SI 2171\/2-2"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2025,1]]},"DOI":"10.1007\/s10664-024-10561-6","type":"journal-article","created":{"date-parts":[[2024,11,30]],"date-time":"2024-11-30T03:52:45Z","timestamp":1732938765000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["\u201cOk Pal, we have to code that now\u201d: interaction patterns of programming beginners with a conversational chatbot"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6204-2095","authenticated-orcid":false,"given":"Alina","family":"Mailach","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dominik","family":"Gorgosch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Norbert","family":"Siegmund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Janet","family":"Siegmund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,11,30]]},"reference":[{"key":"10561_CR1","unstructured":"Barke S, James MB, Polikarpova N (2022) Grounded copilot: how programmers interact with code-generating models. http:\/\/arxiv.org\/abs\/2206.15000. arXiv:2206.15000"},{"key":"10561_CR2","doi-asserted-by":"publisher","unstructured":"Becker BA (2016) An effective approach to enhancing compiler error messages. In: Proceedings of the 47th ACM technical symposium on computing science education. ACM, Memphis Tennessee USA, pp 126\u2013131. https:\/\/doi.org\/10.1145\/2839509.2844584","DOI":"10.1145\/2839509.2844584"},{"key":"10561_CR3","doi-asserted-by":"publisher","unstructured":"Becker BA, Denny P, Finnie-Ansley J, Luxton-Reilly A, Prather J, Santos EA (2023) Programming is hard - or at least it used to be: educational opportunities and challenges of ai code generation. In: Proceedings of the 54th ACM technical symposium on computer science education, vol\u00a01. ACM, Toronto ON Canada, pp 500\u2013506. https:\/\/doi.org\/10.1145\/3545945.3569759","DOI":"10.1145\/3545945.3569759"},{"key":"10561_CR4","unstructured":"Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler D, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language models are few-shot learners. In: Advances in neural information processing systems, vol\u00a033. Curran Associates, Inc., pp 1877\u20131901"},{"key":"10561_CR5","doi-asserted-by":"crossref","unstructured":"Choudhuri R, Liu D, Steinmacher I, Gerosa M, Sarma A (2024) How far are we? the triumphs and trials of generative ai in learning software engineering. In: Proceedings of the IEEE\/ACM 46th international conference on software engineering, pp 1\u201313","DOI":"10.1145\/3597503.3639201"},{"key":"10561_CR6","doi-asserted-by":"publisher","unstructured":"Denny P, Becker BA, Leinonen J, Prather J (2023) Chat overflow: artificially intelligent models for computing education - renAIssance or apocAIypse? In: Proceedings of the 2023 conference on innovation and technology in computer science education, ITiCSE 2023, vol\u00a01. ACM, New York, NY, USA, pp 3\u20134. https:\/\/doi.org\/10.1145\/3587102.3588773","DOI":"10.1145\/3587102.3588773"},{"key":"10561_CR7","doi-asserted-by":"crossref","unstructured":"Denny P, Kumar V, Giacaman N (2023) Conversing with copilot: exploring prompt engineering for solving CS1 problems using natural language. In: Proceedings of the 54th ACM technical symposium on computer science education, SIGCSE 2023, vol\u00a01. ACM, pp 1136\u20131142","DOI":"10.1145\/3545945.3569823"},{"key":"10561_CR8","unstructured":"Denny P, Prather J, Becker BA, Finnie-Ansley J, Hellas A, Leinonen J, Luxton-Reilly A, Reeves BN, Santos EA, Sarsa S (2023) Computing education in the era of generative AI. http:\/\/arxiv.org\/abs\/2306.02608"},{"key":"10561_CR9","doi-asserted-by":"publisher","unstructured":"Finnie-Ansley J, Denny P, Luxton-Reilly A, Santos EA, Prather J, Becker BA (2023) My AI wants to know if this will be on the exam: testing OpenAI\u2019s codex on CS2 programming exercises. In: Proceedings of the 25th australasian computing education conference, ACE \u201923. ACM, New York, NY, USA, pp 97\u2013104. https:\/\/doi.org\/10.1145\/3576123.3576134","DOI":"10.1145\/3576123.3576134"},{"key":"10561_CR10","doi-asserted-by":"crossref","unstructured":"Frankford E, Sauerwein C, Bassner P, Krusche S, Breu R (2024) Ai-tutoring in software engineering education. In: Proceedings of the 45th international conference on software engineering: software engineering education and training, ICSE-SEET \u201923. ACM","DOI":"10.1145\/3639474.3640061"},{"key":"10561_CR11","doi-asserted-by":"publisher","unstructured":"Kazemitabaar M, Chow J, Ma CKT, Ericson BJ, Weintrop D, Grossman T (2023) Studying the effect of ai code generators on supporting novice learners in introductory programming. In: Proceedings of the 2023 CHI conference on human factors in computing systems. ACM, Hamburg Germany, pp 1\u201323. https:\/\/doi.org\/10.1145\/3544548.3580919","DOI":"10.1145\/3544548.3580919"},{"key":"10561_CR12","doi-asserted-by":"crossref","unstructured":"Kazemitabaar M, Hou X, Henley A, Ericson BJ, Weintrop D, Grossman T (2023) How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment. In: Proceedings of the 23rd Koli calling international conference on computing education research, pp 1\u201312","DOI":"10.1145\/3631802.3631806"},{"issue":"1","key":"10561_CR13","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1007\/s10639-022-11177-3","volume":"28","author":"MA Kuhail","year":"2023","unstructured":"Kuhail MA, Alturki N, Alramlawi S, Alhejori K (2023) Interacting with educational chatbots: a systematic review. Educ Inf Technol 28(1):973\u20131018","journal-title":"Educ Inf Technol"},{"key":"10561_CR14","doi-asserted-by":"publisher","unstructured":"Leinonen J, Denny P, MacNeil S, Sarsa S, Bernstein S, Kim J, Tran A, Hellas A (2023) Comparing code explanations created by students and large language models. In: Proceedings of the 2023 conference on innovation and technology in computer science education, ITiCSE 2023, vol\u00a01. Association for Computing Machinery, New York, NY, USA, pp 124\u2013130. https:\/\/doi.org\/10.1145\/3587102.3588785","DOI":"10.1145\/3587102.3588785"},{"key":"10561_CR15","doi-asserted-by":"publisher","unstructured":"Leinonen J, Hellas A, Sarsa S, Reeves B, Denny P, Prather J, Becker BA (2023) Using large language models to enhance programming error messages. In: Proceedings of the 54th ACM technical symposium on computer science education, vol 1. ACM, Toronto ON Canada, pp 563\u2013569. https:\/\/doi.org\/10.1145\/3545945.3569770","DOI":"10.1145\/3545945.3569770"},{"key":"10561_CR16","doi-asserted-by":"crossref","unstructured":"Liang JT, Yang C, Myers BA (2024) A large-scale survey on the usability of AI programming assistants: successes and challenges. In: 2024 IEEE\/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, pp 605\u2013617","DOI":"10.1145\/3597503.3608128"},{"key":"10561_CR17","doi-asserted-by":"publisher","unstructured":"Liu R, Zenke C, Liu C, Holmes A, Thornton P, Malan DJ (2024) Teaching CS50 with AI: leveraging generative artificial intelligence in computer science education. In: Proceedings of the 55th ACM technical symposium on computer science education, vol 1, SIGCSE 2024. ACM, New York, NY, USA, p 7. https:\/\/doi.org\/10.1145\/3626252.3630938","DOI":"10.1145\/3626252.3630938"},{"key":"10561_CR18","doi-asserted-by":"publisher","unstructured":"MacNeil S, Kim J, Leinonen J, Denny P, Bernstein S, Becker BA, Wermelinger M, Hellas A, Tran A, Sarsa S, Prather J, Kumar V (2022) The implications of large language models for CS teachers and students. In: Proceedings of the 54th ACM technical symposium on computer science education vol 2. ACM, Toronto ON Canada, pp 1255\u20131255. https:\/\/doi.org\/10.1145\/3545947.3573358","DOI":"10.1145\/3545947.3573358"},{"key":"10561_CR19","doi-asserted-by":"publisher","unstructured":"MacNeil S, Tran A, Hellas A, Kim J, Sarsa S, Denny P, Bernstein S, Leinonen J (2023) Experiences from using code explanations generated by large language models in a web software development E-Book. In: Proceedings of the 54th ACM technical symposium on computer science education, vol 1, SIGCSE 2023. ACM, pp 931\u2013937. https:\/\/doi.org\/10.1145\/3545945.3569785","DOI":"10.1145\/3545945.3569785"},{"key":"10561_CR20","unstructured":"Meyer B (2022) What do ChatGPT and AI-based automatic program generation mean for the future of software. https:\/\/cacm.acm.org\/blogs\/blog-cacm\/268103-what-do-chatgpt-and-ai-based-automatic-program-generation-mean-for_the-future-of-software\/fulltext"},{"key":"10561_CR21","doi-asserted-by":"crossref","unstructured":"Mozannar H, Bansal G, Fourney A, Horvitz E (2023) Reading between the lines: modeling user behavior and costs in AI-assisted programming","DOI":"10.1145\/3613904.3641936"},{"key":"10561_CR22","doi-asserted-by":"crossref","unstructured":"Nam D, Macvean A, Hellendoorn V, Vasilescu B, Myers B (2024) Using an llm to help with code understanding. In: 2024 IEEE\/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, pp 881\u2013881","DOI":"10.1145\/3597503.3639187"},{"key":"10561_CR23","doi-asserted-by":"publisher","unstructured":"Ouh EL, Gan BKS, Jin\u00a0Shim K, Wlodkowski S (2023) ChatGPT, can you generate solutions for my coding exercises? an evaluation on its effectiveness in an undergraduate Java programming course. In: Proceedings of the 2023 conference on innovation and technology in computer science education, vol 1, ITiCSE 2023. Association for Computing Machinery, New York, NY, USA, pp 54\u201360. https:\/\/doi.org\/10.1145\/3587102.3588794","DOI":"10.1145\/3587102.3588794"},{"key":"10561_CR24","doi-asserted-by":"crossref","unstructured":"Perry N, Srivastava M, Kumar D, Boneh D (2023) Do users write more insecure code with ai assistants? In: Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, pp 2785\u20132799","DOI":"10.1145\/3576915.3623157"},{"key":"10561_CR25","doi-asserted-by":"publisher","unstructured":"Prather J, Denny P, Leinonen J, Becker BA, Albluwi I, Caspersen ME, Craig M, Keuning H, Kiesler N, Kohn T, Luxton-Reilly A, MacNeil S, Petersen A, Pettit R, Reeves BN, Savelka J (2023) Transformed by transformers: navigating the AI coding revolution for computing education: an ITiCSE working group conducted by humans. In: Proceedings of the 2023 conference on innovation and technology in computer science education, vol 2. ACM, Turku Finland, pp 561\u2013562. https:\/\/doi.org\/10.1145\/3587103.3594206","DOI":"10.1145\/3587103.3594206"},{"key":"10561_CR26","doi-asserted-by":"publisher","unstructured":"Prather J, Reeves BN, Denny P, Becker BA, Leinonen J, Luxton-Reilly A, Powell G, Finnie-Ansley J, Santos EA (2023) \u201cIt\u2019s Weird That It Knows What I Want\u201d: usability and interactions with copilot for novice programmers. ACM Trans Comput-Hum Interact 31(1). https:\/\/doi.org\/10.1145\/3617367","DOI":"10.1145\/3617367"},{"key":"10561_CR27","doi-asserted-by":"crossref","unstructured":"Robins AV (2019) 12 novice programmers and introductory programming. The Cambridge handbook of computing education research, pp 327\u2013376","DOI":"10.1017\/9781108654555.013"},{"key":"10561_CR28","doi-asserted-by":"publisher","unstructured":"Ross SI, Martinez F, Houde S, Muller M, Weisz JD (2023) The programmer\u2019s assistant: conversational interaction with a large language model for software development. In: Proceedings of the 28th international conference on intelligent user interfaces, IUI \u201923. ACM, pp 491\u2013514. https:\/\/doi.org\/10.1145\/3581641.3584037","DOI":"10.1145\/3581641.3584037"},{"key":"10561_CR29","unstructured":"Sandoval G, Pearce H, Nys T, Karri R, Garg S, Dolan-Gavitt B (2023) Lost at C: A user study on the security implications of large language model code assistants. In: 32nd USENIX Security Symposium (USENIX Security 23). USENIX Association, Anaheim, CA, pp 2205\u20132222"},{"issue":"4","key":"10561_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3628162","volume":"23","author":"A Shoufan","year":"2023","unstructured":"Shoufan A (2023) Can students without prior knowledge use chatgpt to answer test questions? an empirical study. ACM Trans Comput Educ 23(4):1\u201329","journal-title":"ACM Trans Comput Educ"},{"key":"10561_CR31","doi-asserted-by":"publisher","first-page":"1299","DOI":"10.1007\/s10664-013-9286-4","volume":"19","author":"J Siegmund","year":"2014","unstructured":"Siegmund J, K\u00e4stner C, Liebig J, Apel S, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19:1299\u20131334","journal-title":"Empir Softw Eng"},{"key":"10561_CR32","doi-asserted-by":"publisher","unstructured":"Vaithilingam P, Zhang T, Glassman EL (2022) Expectation vs. experience: evaluating the usability of code generation tools powered by large language models. In: Extended abstracts of the 2022 CHI conference on human factors in computing systems, CHI EA \u201922. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3491101.3519665","DOI":"10.1145\/3491101.3519665"},{"key":"10561_CR33","doi-asserted-by":"publisher","unstructured":"Verleger M, Pembridge J (2018) A pilot study integrating an AI-driven chatbot in an introductory programming course. In: 2018 IEEE Frontiers in Education Conference (FIE), pp 1\u20134. https:\/\/doi.org\/10.1109\/FIE.2018.8659282","DOI":"10.1109\/FIE.2018.8659282"},{"key":"10561_CR34","first-page":"24824","volume":"35","author":"J Wei","year":"2022","unstructured":"Wei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, Le QV, Zhou D (2022) Chain of thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst 35:24824\u201324837","journal-title":"Adv Neural Inf Process Syst"},{"key":"10561_CR35","doi-asserted-by":"publisher","unstructured":"Weisz JD, Muller M, Ross SI, Martinez F, Houde S, Agarwal M, Talamadupula K, Richards JT (2022) Better together? an evaluation of AI-supported code translation. In: 27th International conference on intelligent user interfaces, IUI \u201922. ACM, New York, NY, USA, pp 369\u2013391. https:\/\/doi.org\/10.1145\/3490099.3511157","DOI":"10.1145\/3490099.3511157"},{"key":"10561_CR36","doi-asserted-by":"publisher","unstructured":"Wermelinger M (2023) Using GitHub copilot to solve simple programming problems. In: Proceedings of the 54th ACM technical symposium on computer science education, vol 1. ACM, Toronto ON Canada, pp 172\u2013178. https:\/\/doi.org\/10.1145\/3545945.3569830","DOI":"10.1145\/3545945.3569830"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10561-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10561-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10561-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T14:16:56Z","timestamp":1736259416000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10561-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,30]]},"references-count":36,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10561"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10561-6","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,30]]},"assertion":[{"value":"30 September 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 November 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"34"}}