{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T23:05:09Z","timestamp":1776121509952,"version":"3.50.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,12,11]],"date-time":"2023-12-11T00:00:00Z","timestamp":1702252800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2023,12,31]]},"abstract":"<jats:p>With the immense interest in ChatGPT worldwide, education has seen a mix of both excitement and skepticism. To properly evaluate its impact on education, it is crucial to understand how far it can help students without prior knowledge answer assessment questions. This study aims to address this question as well as the impact of the question type. We conducted multiple experiments with computer engineering students (experiment group:<jats:italic>n<\/jats:italic>=41 to 56), who were asked to use ChatGPT to answer previous test questions before learning about the related topics. Their scores were then compared with the scores of previous-term students who answered the same questions in a quiz or exam setting (control group:<jats:italic>n<\/jats:italic>=24 to 61). The results showed a wide range of effect sizes, from -2.55 to 1.23, depending on the question type and content. The experiment group performed best answering code analysis and conceptual questions but struggled with code completion and questions that involved images. However, the performance in code generation tasks was inconsistent. Overall, the ChatGPT group\u2019s answers lagged slightly behind the control group\u2019s answers with an effect size of -0.16. We conclude that ChatGPT, at least in the field of this study, is not yet ready to rely on by students who do not have sufficient background to evaluate generated answers. We suggest that educators try using ChatGPT and educate students on effective questioning techniques and how to assess the generated responses. This study provides insights into the capabilities and limitations of ChatGPT in education and informs future research and development.<\/jats:p>","DOI":"10.1145\/3628162","type":"journal-article","created":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T22:11:22Z","timestamp":1697667082000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":44,"title":["Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study"],"prefix":"10.1145","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3968-8637","authenticated-orcid":false,"given":"Abdulhadi","family":"Shoufan","sequence":"first","affiliation":[{"name":"Electrical Engineering and Computer Science Department, Center for Cyber-Physical Systems, Khalifa University, UAE"}]}],"member":"320","published-online":{"date-parts":[[2023,12,11]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"100006","article-title":"Chatbots: History, technology, and applications","volume":"2","author":"Adamopoulou Eleni","year":"2020","unstructured":"Eleni Adamopoulou and Lefteris Moussiades. 2020. Chatbots: History, technology, and applications. Mach. Learn. Appl. 2 (2020), 100006.","journal-title":"Mach. Learn. Appl."},{"key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/10494820.2022.2075014","article-title":"Understanding the impact of knowledge management factors on the sustainable use of AI-based chatbots for educational purposes using a hybrid SEM-ANN approach","author":"Al-Sharafi Mohammed A.","year":"2022","unstructured":"Mohammed A. Al-Sharafi, Mostafa Al-Emran, Mohammad Iranmanesh, Noor Al-Qaysi, Noorminshah A. Iahad, and Ibrahim Arpaci. 2022. Understanding the impact of knowledge management factors on the sustainable use of AI-based chatbots for educational purposes using a hybrid SEM-ANN approach. Interact. Learn. Environ. (2022), 1\u201320.","journal-title":"Interact. Learn. Environ."},{"key":"e_1_3_3_4_2","doi-asserted-by":"crossref","unstructured":"Yejin Bang Samuel Cahyawijaya Nayeon Lee Wenliang Dai Dan Su Bryan Wilie Holy Lovenia Ziwei Ji Tiezheng Yu Willy Chung et\u00a0al. 2023. A multitask multilingual multimodal evaluation of ChatGPT on reasoning hallucination and interactivity. Retrieved from https:\/\/arXiv:2302.04023","DOI":"10.18653\/v1\/2023.ijcnlp-main.45"},{"key":"e_1_3_3_5_2","first-page":"500","volume-title":"Proceedings of the 54th ACM Technical Symposium on Computer Science Education","author":"Becker Brett A.","year":"2023","unstructured":"Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming is hard-or at least it used to be: Educational opportunities and challenges of AI code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 500\u2013506."},{"issue":"2","key":"e_1_3_3_6_2","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/TLT.2017.2682084","article-title":"A tool for introducing computer science with automatic formative assessment","volume":"11","author":"Benotti Luciana","year":"2017","unstructured":"Luciana Benotti, Mara Cecilia Martnez, and Fernando Schapachnik. 2017. A tool for introducing computer science with automatic formative assessment. IEEE Trans. Learn. Technol. 11, 2 (2017), 179\u2013192.","journal-title":"IEEE Trans. Learn. Technol."},{"key":"e_1_3_3_7_2","first-page":"1","article-title":"Artificial intelligence (AI) student assistants in the classroom: Designing chatbots to support student success","author":"Chen Yu","year":"2022","unstructured":"Yu Chen, Scott Jensen, Leslie J. Albert, Sambhav Gupta, and Terri Lee. 2022. Artificial intelligence (AI) student assistants in the classroom: Designing chatbots to support student success. Info. Syst. Front. 25, 1 (2022), 1\u201322.","journal-title":"Info. Syst. Front."},{"key":"e_1_3_3_8_2","first-page":"1136","volume-title":"Proceedings of the 54th ACM Technical Symposium on Computer Science Education","author":"Denny Paul","year":"2023","unstructured":"Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 1136\u20131142."},{"key":"e_1_3_3_9_2","doi-asserted-by":"crossref","unstructured":"M. El-Abd. 2023. A review of embedded systems education in the arduino age: lessons learned and future directions. International Journal of Engineering Pedagogy 7 2 (2023) 79\u201393. Retrieved November 2 2023 from https:\/\/www.learntechlib.org\/p\/207404\/","DOI":"10.3991\/ijep.v7i2.6845"},{"issue":"1","key":"e_1_3_3_10_2","first-page":"1","article-title":"The impact of a virtual teaching assistant (chatbot) on students\u2019 learning in Ghanaian higher education","volume":"19","author":"Essel Harry Barton","year":"2022","unstructured":"Harry Barton Essel, Dimitrios Vlachopoulos, Akosua Tachie-Menson, Esi Eduafua Johnson, and Papa Kwame Baah. 2022. The impact of a virtual teaching assistant (chatbot) on students\u2019 learning in Ghanaian higher education. Int. J. Edu. Technol. Higher Edu. 19, 1 (2022), 1\u201319.","journal-title":"Int. J. Edu. Technol. Higher Edu."},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/3511861.3511863","volume-title":"Proceedings of the Australasian Computing Education Conference","author":"Finnie-Ansley James","year":"2022","unstructured":"James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The robots are coming: Exploring the implications of openai codex on introductory programming. In Proceedings of the Australasian Computing Education Conference. 10\u201319."},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1145\/3576123.3576134","volume-title":"Proceedings of the 25th Australasian Computing Education Conference","author":"Finnie-Ansley James","year":"2023","unstructured":"James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI wants to know if this will be on the exam: testing OpenAI\u2019s codex on CS2 programming exercises. In Proceedings of the 25th Australasian Computing Education Conference. 97\u2013104."},{"issue":"1","key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the united states medical licensing examination? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson Aidan","year":"2023","unstructured":"Aidan Gilson, Conrad W. Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et\u00a0al. 2023. How does ChatGPT perform on the united states medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med. Edu. 9, 1 (2023), e45312.","journal-title":"JMIR Med. Edu."},{"key":"e_1_3_3_14_2","first-page":"946","volume-title":"Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications: Web, Artificial Intelligence, and Network Applications (WAINA\u201919) 33","author":"Hussain Shafquat","year":"2019","unstructured":"Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents\/chatbots classification and design techniques. In Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications: Web, Artificial Intelligence, and Network Applications (WAINA\u201919) 33. Springer, 946\u2013956."},{"key":"e_1_3_3_15_2","first-page":"109","article-title":"Chatbots as cognitive, educational, advisory & coaching systems","volume":"30","author":"Karyotaki Maria","year":"2022","unstructured":"Maria Karyotaki, Athanasios Drigas, and Charalabos Skianis. 2022. Chatbots as cognitive, educational, advisory & coaching systems. Technium Soc. Sci. J. 30 (2022), 109\u2013126.","journal-title":"Technium Soc. Sci. J."},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","unstructured":"Enkelejda Kasneci Kathrin Sessler Stefan K\u00fcchemann Maria Bannert Daryna Dementieva Frank Fischer Urs Gasser Georg Groh Stephan G\u00fcnnemann Eyke H\u00fcllermeier Stephan Krusche Gitta Kutyniok Tilman Michaeli Claudia Nerdel J\u00fcrgen Pfeffer Oleksandra Poquet Michael Sailer Albrecht Schmidt Tina Seidel Matthias Stadler Jochen Weller Jochen Kuhn and Gjergji Kasneci. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences 103 (2023) 102274. DOI:10.1016\/j.lindif.2023.102274","DOI":"10.1016\/j.lindif.2023.102274"},{"key":"e_1_3_3_17_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Kazemitabaar Majeed","year":"2023","unstructured":"Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI code generators on supporting novice learners in introductory programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1\u201323."},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","first-page":"100364","DOI":"10.1016\/j.cosrev.2021.100364","article-title":"A systematic literature review on prototyping with Arduino: Applications, challenges, advantages, and limitations","volume":"40","author":"Kondaveeti Hari Kishan","year":"2021","unstructured":"Hari Kishan Kondaveeti, Nandeesh Kumar Kumaravelu, Sunny Dayal Vanambathina, Sudha Ellison Mathe, and Suseela Vappangi. 2021. A systematic literature review on prototyping with Arduino: Applications, challenges, advantages, and limitations. Comput. Sci. Rev. 40 (2021), 100364.","journal-title":"Comput. Sci. Rev."},{"key":"e_1_3_3_19_2","first-page":"1","article-title":"Interacting with educational chatbots: A systematic review","author":"Kuhail Mohammad Amin","year":"2022","unstructured":"Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. 2022. Interacting with educational chatbots: A systematic review. Edu. Info. Technol. 28, 1 (2022), 1\u201346.","journal-title":"Edu. Info. Technol."},{"key":"e_1_3_3_20_2","first-page":"1339","volume-title":"Proceedings of the 54th ACM Technical Symposium on Computer Science Education","author":"Kumar Harsh","year":"2022","unstructured":"Harsh Kumar, Kunzhi Yu, Andrew Chung, Jiakai Shi, and Joseph Jay Williams. 2022. Exploring the potential of chatbots to provide mental well-being support for computer science students. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 1339\u20131339."},{"issue":"2","key":"e_1_3_3_21_2","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"Kung Tiffany H.","year":"2023","unstructured":"Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepa\u00f1o, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et\u00a0al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Dig. Health 2, 2 (2023), e0000198.","journal-title":"PLOS Dig. Health"},{"key":"e_1_3_3_22_2","doi-asserted-by":"crossref","unstructured":"Sam Lau and Philip Guo. 2023. From \u201cBan it till we understand it\u201d to \u201cResistance is futile\u201d: How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 106\u2013121.","DOI":"10.1145\/3568813.3600138"},{"key":"e_1_3_3_23_2","first-page":"348","volume-title":"Proceedings of the 21st International Conference on Human-Computer Interaction (HCII\u201919)","author":"Lee Keeheon","year":"2019","unstructured":"Keeheon Lee, Jeongwon Jo, Jinyoung Kim, and Younah Kang. 2019. Can chatbots help reduce the workload of administrative officers?-Implementing and deploying FAQ chatbot service in a university. In Proceedings of the 21st International Conference on Human-Computer Interaction (HCII\u201919). Springer, 348\u2013354."},{"key":"e_1_3_3_24_2","first-page":"563","volume-title":"Proceedings of the 54th ACM Technical Symposium on Computer Science Education","author":"Leinonen Juho","year":"2023","unstructured":"Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 563\u2013569."},{"issue":"6624","key":"e_1_3_3_25_2","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1126\/science.abq1158","article-title":"Competition-level code generation with alphacode","volume":"378","author":"Li Yujia","year":"2022","unstructured":"Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\u00e9mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago et\u00a0al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092\u20131097.","journal-title":"Science"},{"key":"e_1_3_3_26_2","first-page":"931","volume-title":"Proceedings of the 54th ACM Technical Symposium on Computer Science Education","author":"MacNeil Stephen","year":"2023","unstructured":"Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 931\u2013937."},{"key":"e_1_3_3_27_2","first-page":"37","volume-title":"Proceedings of the ACM Conference on International Computing Education Research","author":"MacNeil Stephen","year":"2022","unstructured":"Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the GPT-3 large language model. In Proceedings of the ACM Conference on International Computing Education Research. 37\u201339."},{"key":"e_1_3_3_28_2","unstructured":"Beth McMurtrie. 2023. Teaching Experts Are Worried About ChatGPT but Not for the Reasons You Think. Retrieved from https:\/\/www.chronicle.com\/article\/ai-and-the-future-of-undergraduate-writing"},{"key":"e_1_3_3_29_2","first-page":"365","volume-title":"Proceedings of the20th International Conference in Artificial Intelligence in Education (AIED\u201919)","author":"Ndukwe Ifeanyi G.","year":"2019","unstructured":"Ifeanyi G. Ndukwe, Ben K. Daniel, and Chukwudi E. Amadi. 2019. A machine learning grading system using chatbots. In Proceedings of the20th International Conference in Artificial Intelligence in Education (AIED\u201919). Springer, 365\u2013368."},{"key":"e_1_3_3_30_2","doi-asserted-by":"crossref","first-page":"104661","DOI":"10.1016\/j.compedu.2022.104661","article-title":"Role design considerations of conversational agents to facilitate discussion and systems thinking","volume":"192","author":"Nguyen Ha","year":"2023","unstructured":"Ha Nguyen. 2023. Role design considerations of conversational agents to facilitate discussion and systems thinking. Comput. Edu. 192 (2023), 104661.","journal-title":"Comput. Edu."},{"key":"e_1_3_3_31_2","first-page":"100036","article-title":"NEU-chatbot: Chatbot for admission of national economics university","volume":"2","author":"Nguyen Trung Thanh","year":"2021","unstructured":"Trung Thanh Nguyen, Anh Duc Le, Ha Thanh Hoang, and Tuan Nguyen. 2021. NEU-chatbot: Chatbot for admission of national economics university. Comput. Edu.: Artific. Intell. 2 (2021), 100036.","journal-title":"Comput. Edu.: Artific. Intell."},{"key":"e_1_3_3_32_2","first-page":"100033","article-title":"Chatbots applications in education: A systematic review","volume":"2","author":"Okonkwo Chinedu Wilfred","year":"2021","unstructured":"Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. 2021. Chatbots applications in education: A systematic review. Comput. Edu.: Artific. Intell. 2 (2021), 100033.","journal-title":"Comput. Edu.: Artific. Intell."},{"key":"e_1_3_3_33_2","unstructured":"Brian Owens. 2023. How Nature readers are using ChatGPT. Retrieved from https:\/\/www.nature.com\/articles\/d41586-023-00500-8"},{"issue":"6","key":"e_1_3_3_34_2","doi-asserted-by":"crossref","first-page":"1549","DOI":"10.1002\/cae.22326","article-title":"Rediscovering the use of chatbots in education: A systematic literature review","volume":"28","author":"P\u00e9rez Jos\u00e9 Quiroga","year":"2020","unstructured":"Jos\u00e9 Quiroga P\u00e9rez, Thanasis Daradoumis, and Joan Manuel Marqu\u00e8s Puig. 2020. Rediscovering the use of chatbots in education: A systematic literature review. Comput. Appl. Eng. Edu. 28, 6 (2020), 1549\u20131565.","journal-title":"Comput. Appl. Eng. Edu."},{"issue":"1","key":"e_1_3_3_35_2","article-title":"ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?","volume":"6","author":"Rudolph J\u00fcrgen","year":"2023","unstructured":"J\u00fcrgen Rudolph, Samson Tan, and Shannon Tan. 2023. ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? J. Appl. Learn. Teach. 6, 1 (2023).","journal-title":"J. Appl. Learn. Teach."},{"key":"e_1_3_3_36_2","doi-asserted-by":"crossref","first-page":"e12843","DOI":"10.1016\/j.heliyon.2023.e12843","article-title":"Perceived satisfaction of university students with the use of chatbots as a tool for self-regulated learning","author":"S\u00e1iz-Manzanares Mar\u00eda Consuelo","year":"2023","unstructured":"Mar\u00eda Consuelo S\u00e1iz-Manzanares, Ra\u00fal Marticorena-S\u00e1nchez, Luis Jorge Mart\u00edn-Ant\u00f3n, Irene Gonz\u00e1lez D\u00edez, and Leandro Almeida. 2023. Perceived satisfaction of university students with the use of chatbots as a tool for self-regulated learning. Heliyon 9, 1 (2023), e12843.","journal-title":"Heliyon"},{"key":"e_1_3_3_37_2","first-page":"27","volume-title":"Proceedings of the ACM Conference on International Computing Education Research","author":"Sarsa Sami","year":"2022","unstructured":"Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the ACM Conference on International Computing Education Research. 27\u201343."},{"key":"e_1_3_3_38_2","unstructured":"Shipra Sharma and Balwinder Sodhi. 2023. Calculating originality of LLM assisted source code. Retrieved from https:\/\/arXiv:2307.04492"},{"issue":"4","key":"e_1_3_3_39_2","doi-asserted-by":"crossref","first-page":"1119","DOI":"10.1007\/s40593-022-00300-7","article-title":"Automated essay writing: An AIED opinion","volume":"32","author":"Sharples Mike","year":"2022","unstructured":"Mike Sharples. 2022. Automated essay writing: An AIED opinion. Int. J. Artific. Intell. Edu. 32, 4 (2022), 1119\u20131126.","journal-title":"Int. J. Artific. Intell. Edu."},{"issue":"4","key":"e_1_3_3_40_2","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1109\/TE.2020.2989921","article-title":"Lecture-free classroom: Fully active learning on Moodle","volume":"63","author":"Shoufan Abdulhadi","year":"2020","unstructured":"Abdulhadi Shoufan. 2020. Lecture-free classroom: Fully active learning on Moodle. IEEE Trans. Edu. 63, 4 (2020), 314\u2013321.","journal-title":"IEEE Trans. Edu."},{"key":"e_1_3_3_41_2","doi-asserted-by":"crossref","first-page":"41104","DOI":"10.1109\/ACCESS.2021.3065248","article-title":"Active distance learning of embedded systems","volume":"9","author":"Shoufan Abdulhadi","year":"2021","unstructured":"Abdulhadi Shoufan. 2021. Active distance learning of embedded systems. IEEE Access 9 (2021), 41104\u201341122.","journal-title":"IEEE Access"},{"key":"e_1_3_3_42_2","doi-asserted-by":"crossref","unstructured":"A. Shoufan. 2023. Exploring students\u2019 perceptions of ChatGPT: Thematic analysis and follow-up survey. IEEE Access 11 (2023) 38805\u201338818. DOI:10.1109\/ACCESS.2023.3268224","DOI":"10.1109\/ACCESS.2023.3268224"},{"key":"e_1_3_3_43_2","first-page":"1","volume-title":"Proceedings of the IEEE Frontiers in Education Conference (FIE\u201920)","author":"Sindre Guttorm","year":"2020","unstructured":"Guttorm Sindre. 2020. Code writing vs code completion puzzles: Analyzing questions in an E-Exam. In Proceedings of the IEEE Frontiers in Education Conference (FIE\u201920). IEEE, 1\u20139."},{"key":"e_1_3_3_44_2","doi-asserted-by":"crossref","first-page":"103862","DOI":"10.1016\/j.compedu.2020.103862","article-title":"Chatbots for learning: A review of educational chatbots for the Facebook Messenger","volume":"151","author":"Smutny Pavel","year":"2020","unstructured":"Pavel Smutny and Petra Schreiberova. 2020. Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Comput. Edu. 151 (2020), 103862.","journal-title":"Comput. Edu."},{"key":"e_1_3_3_45_2","unstructured":"Teo Susnjak. 2022. ChatGPT: The end of online exam integrity? Retrieved from https:\/\/arXiv:2212.09292"},{"key":"e_1_3_3_46_2","first-page":"563","article-title":"Using large language models to automatically identify programming concepts in code snippets","volume":"1","author":"Tran Andrew","year":"2023","unstructured":"Andrew Tran, Linxuan Li, Egi Rama, Kenneth Angelikas, and Stephen MacNeil. 2023. Using large language models to automatically identify programming concepts in code snippets. Sci. Edu. 1 (2023), 563\u2013569.","journal-title":"Sci. Edu."},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","unstructured":"Jos\u00e9 Fidel Urquiza-Yllescas Sonia Mendoza Jos\u00e9 Rodr\u00edguez and Luis Mart\u00edn S\u00e1nchez-Adame. 2022. An approach to the classification of educational chatbots. Journal of Intelligent & Fuzzy Systems 43 4 (2022) 5095\u20135107.","DOI":"10.3233\/JIFS-213275"},{"key":"e_1_3_3_48_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts","author":"Vaithilingam Priyan","year":"2022","unstructured":"Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1\u20137."},{"key":"e_1_3_3_49_2","unstructured":"Eva A. M. van Dis Johan Bollen Willem Zuidema Robert van Rooij and Claudi L. Bockting. 2023. ChatGPT: Five priorities for research. Retrieved from https:\/\/www.nature.com\/articles\/d41586-023-00288-7"},{"key":"e_1_3_3_50_2","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1145\/3545945.3569830","volume-title":"Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1","author":"Wermelinger Michel","year":"2023","unstructured":"Michel Wermelinger. 2023. Using GitHub Copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 172\u2013178."},{"key":"e_1_3_3_51_2","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. Retrieved from https:\/\/arXiv:2302.11382"},{"key":"e_1_3_3_52_2","first-page":"79","volume-title":"Proceedings of the 3rd International Conference on Education and E-Learning","author":"Yang Shanshan","year":"2019","unstructured":"Shanshan Yang and Chris Evans. 2019. Opportunities and challenges in using AI chatbots in higher education. In Proceedings of the 3rd International Conference on Education and E-Learning. 79\u201383."}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3628162","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3628162","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:49:02Z","timestamp":1750286942000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3628162"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,11]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12,31]]}},"alternative-id":["10.1145\/3628162"],"URL":"https:\/\/doi.org\/10.1145\/3628162","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,11]]},"assertion":[{"value":"2023-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-12","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}