{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T08:16:15Z","timestamp":1771229775546,"version":"3.50.1"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,8,16]],"date-time":"2025-08-16T00:00:00Z","timestamp":1755302400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,16]],"date-time":"2025-08-16T00:00:00Z","timestamp":1755302400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Intell Inf Syst"],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Recent advances in artificial intelligence (AI), machine learning (ML), and natural language processing (NLP) have led to the development of a new generation of Large Language Models (LLMs) trained on massive amounts of data. Commercial applications (e.g., ChatGPT) have made this available to the general public, enabling the use of LLMs to produce high-quality texts for academic and professional purposes. Educational institutions are increasingly aware of students\u2019 use of AI-generated content and are researching its impact and potential misuse. Computer Science (CS) and related fields are particularly affected, as LLMs can also generate programming code in various languages. To understand the potential impact of publicly available LLMs in CS education, we extend our previously introduced  (Raihan et al. 2024), a framework comprising hundreds of programming exercise prompts and multiple-choice questions from introductory CS and programming courses. We provide experimental results on , evaluating the performance of several LLMs in generating Python code and answering basic computer science and programming questions, offering insights into the implications of this technology for CS education.<\/jats:p>","DOI":"10.1007\/s10844-025-00968-y","type":"journal-article","created":{"date-parts":[[2025,8,16]],"date-time":"2025-08-16T08:42:45Z","timestamp":1755333765000},"page":"239-263","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["On the performance of large language models on introductory programming assignments"],"prefix":"10.1007","volume":"64","author":[{"given":"Nishat","family":"Raihan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dhiman","family":"Goswami","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sadiya Sayara Chowdhury","family":"Puspo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammed Latif","family":"Siddiq","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Newman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tharindu","family":"Ranasinghe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joanna C. S.","family":"Santos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcos","family":"Zampieri","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,8,16]]},"reference":[{"key":"968_CR1","doi-asserted-by":"publisher","unstructured":"Achiam, J.,\u00a0Adler, S.,\u00a0Agarwal, S., et\u00a0al. (2023.) Gpt-4 technical report. arXiv:2303.08774. https:\/\/doi.org\/10.48550\/arXiv.2303.08774 .","DOI":"10.48550\/arXiv.2303.08774"},{"key":"968_CR2","doi-asserted-by":"publisher","unstructured":"Austin, J.,\u00a0Odena, A.,\u00a0Nye, M., et\u00a0al. (2021). Program synthesis with large language models. arXiv:2108.07732. https:\/\/doi.org\/10.48550\/arXiv.2108.07732 .","DOI":"10.48550\/arXiv.2108.07732"},{"key":"968_CR3","doi-asserted-by":"publisher","unstructured":"Black, S.,\u00a0Gao, L.,\u00a0Wang, P., et\u00a0al. (2022). Gpt-neox-20b: An open-source autoregressive language model. arXiv:2204.06745. https:\/\/doi.org\/10.48550\/arXiv.2204.06745 .","DOI":"10.48550\/arXiv.2204.06745"},{"key":"968_CR4","doi-asserted-by":"publisher","unstructured":"Brown, T. B.,\u00a0Mann, B.,\u00a0Ryder, N., et\u00a0al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems. https:\/\/doi.org\/10.48550\/arXiv.2005.14165 .","DOI":"10.48550\/arXiv.2005.14165"},{"key":"968_CR5","doi-asserted-by":"publisher","unstructured":"Chen, M.,\u00a0Tworek, J.,\u00a0Jun, H., et\u00a0al. (2021). Evaluating large language models trained on code. arXiv:2107.03374. https:\/\/doi.org\/10.48550\/arXiv.2107.03374 .","DOI":"10.48550\/arXiv.2107.03374"},{"key":"968_CR6","doi-asserted-by":"publisher","unstructured":"Chowdhery, A.,\u00a0Narang, S.,\u00a0Devlin, J., et\u00a0al. (2022). Palm: Scaling language modeling with pathways. arXiv:2204.02311. https:\/\/doi.org\/10.48550\/arXiv.2204.02311 .","DOI":"10.48550\/arXiv.2204.02311"},{"key":"968_CR7","doi-asserted-by":"publisher","unstructured":"Devlin, J., Chang, M. W.,\u00a0Lee, K., et\u00a0al. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. https:\/\/doi.org\/10.18653\/V1\/N19-1423","DOI":"10.18653\/V1\/N19-1423"},{"key":"968_CR8","doi-asserted-by":"publisher","unstructured":"Dubey, A.,\u00a0Jauhri, A.,\u00a0Pandey, A., et\u00a0al. (2024). The llama 3 herd of models. arXiv:2407.21783. https:\/\/doi.org\/10.48550\/arXiv.2407.21783 .","DOI":"10.48550\/arXiv.2407.21783"},{"key":"968_CR9","doi-asserted-by":"publisher","unstructured":"Feng, Z.,\u00a0Guo, D.,\u00a0Tang, D., et\u00a0al. (2020). Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. https:\/\/doi.org\/10.18653\/v1\/2020.findings-emnlp.139.","DOI":"10.18653\/v1\/2020.findings-emnlp.139."},{"key":"968_CR10","doi-asserted-by":"crossref","unstructured":"Green, C. (1969). Application of theorem proving to problem solving. In Proc. of the 1st Intl. Joint Conf. on Artificial Intelligence. IJCAI\u201969, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. https:\/\/www.ijcai.org\/Proceedings\/69\/Papers\/023.pdf","DOI":"10.21236\/ADA459656"},{"key":"968_CR11","doi-asserted-by":"publisher","unstructured":"Gulwani, S.,\u00a0Polozov, O.,\u00a0Singh, R., et\u00a0al. (2017). Program synthesis. Foundations and Trends\u00ae in Programming Languages,\u00a04(1-2): 1\u2013119. https:\/\/doi.org\/10.1561\/2500000010 .","DOI":"10.1561\/2500000010"},{"key":"968_CR12","doi-asserted-by":"publisher","unstructured":"Guo, D.,\u00a0Ren, S.,\u00a0Lu, S., et\u00a0al. (2020). Graphcodebert: Pre-training code representations with data flow. arXiv:2009.08366. https:\/\/doi.org\/10.48550\/arXiv.2009.08366 .","DOI":"10.48550\/arXiv.2009.08366"},{"key":"968_CR13","doi-asserted-by":"publisher","unstructured":"Halaweh, M. (2023). Chatgpt in education: Strategies for responsible implementation. Contemporary Educational Technology,\u00a015(2). https:\/\/doi.org\/10.30935\/cedtech\/13036","DOI":"10.30935\/cedtech\/13036"},{"key":"968_CR14","doi-asserted-by":"publisher","unstructured":"Haruna-Cooper, L., & Rashid, M. A. (2023). Gpt-4: the future of artificial intelligence in medical school assessments. Journal of the Royal Society of Medicine, 01410768231181251. https:\/\/doi.org\/10.1177\/01410768231181251","DOI":"10.1177\/01410768231181251"},{"key":"968_CR15","doi-asserted-by":"publisher","unstructured":"Iyer, S.,\u00a0Konstas, I.,\u00a0Cheung, A., et\u00a0al. (2018). Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. https:\/\/doi.org\/10.18653\/v1\/D18-1192. PMLR","DOI":"10.18653\/v1\/D18-1192"},{"key":"968_CR16","doi-asserted-by":"publisher","unstructured":"Jiang, A. Q.,\u00a0Sablayrolles, A.,\u00a0Mensch, A., et\u00a0al. (2023). Mistral 7b. arXiv:2310.06825. https:\/\/doi.org\/10.48550\/arXiv.2310.06825","DOI":"10.48550\/arXiv.2310.06825"},{"key":"968_CR17","doi-asserted-by":"publisher","unstructured":"Katz, D. M., Bommarito, M. J., Gao, S., et al. (2023). Gpt-4 passes the bar exam. SSRN. https:\/\/doi.org\/10.2139\/ssrn.4389233","DOI":"10.2139\/ssrn.4389233"},{"key":"968_CR18","doi-asserted-by":"publisher","unstructured":"Kulal, S.,\u00a0Pasupat, P.,\u00a0Chandra, K., et\u00a0al. (2019). Spoc: Search-based pseudocode to code. In Advances in Neural Information Processing Systems. https:\/\/doi.org\/10.48550\/arXiv.1906.04908","DOI":"10.48550\/arXiv.1906.04908"},{"key":"968_CR19","doi-asserted-by":"publisher","unstructured":"Li, R., Allal, L. B.,\u00a0Zi, Y., et\u00a0al. (2023). Starcoder: may the source be with you! arXiv:2305.06161. https:\/\/doi.org\/10.48550\/arXiv.2305.06161","DOI":"10.48550\/arXiv.2305.06161"},{"key":"968_CR20","doi-asserted-by":"publisher","unstructured":"Liu, J., Xia, C. S.,\u00a0Wang, Y., et\u00a0al. (2024). Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems. https:\/\/doi.org\/10.48550\/arXiv.2305.01337 36","DOI":"10.48550\/arXiv.2305.01337"},{"issue":"4","key":"968_CR21","doi-asserted-by":"publisher","first-page":"410","DOI":"10.3390\/educsci13040410","volume":"13","author":"CK Lo","year":"2023","unstructured":"Lo, C. K. (2023). What is the impact of chatgpt on education? a rapid review of the literature. Education Sciences, 13(4), 410. https:\/\/doi.org\/10.3390\/educsci13040410","journal-title":"Education Sciences"},{"key":"968_CR22","doi-asserted-by":"publisher","unstructured":"Lukasczyk, S., &\u00a0Fraser, G. (2022). Pynguin: Automated unit test generation for python. In Proceedings of the ACM\/IEEE 44th International Conference on Software Engineering: Companion Proceedings (pp. 168\u2013172). https:\/\/doi.org\/10.1145\/3510454.3516829","DOI":"10.1145\/3510454.3516829"},{"key":"968_CR23","doi-asserted-by":"publisher","unstructured":"Luo, Z.,\u00a0Xu, C.,\u00a0Zhao, P., et\u00a0al. (2023). Wizardcoder: Empowering code large language models with evol-instruct. arXiv:2306.08568. https:\/\/doi.org\/10.48550\/arXiv.2306.08568","DOI":"10.48550\/arXiv.2306.08568"},{"key":"968_CR24","doi-asserted-by":"publisher","unstructured":"Manna, Z., & Waldinger, R. J. (1971). Toward automatic program synthesis. https:\/\/doi.org\/10.1145\/362566.362568","DOI":"10.1145\/362566.362568"},{"key":"968_CR25","doi-asserted-by":"publisher","unstructured":"Mikolov, T.,\u00a0Sutskever, I.,\u00a0Chen, K., et\u00a0al. (2013). Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. https:\/\/doi.org\/10.48550\/arXiv.1310.4546","DOI":"10.48550\/arXiv.1310.4546"},{"key":"968_CR26","doi-asserted-by":"publisher","unstructured":"Nijkamp, E.,\u00a0Lee, J.,\u00a0Touvron, H., et\u00a0al. (2022). Codegen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. https:\/\/doi.org\/10.48550\/arXiv.2203.13474","DOI":"10.48550\/arXiv.2203.13474"},{"key":"968_CR27","doi-asserted-by":"publisher","unstructured":"OpenAI. (2024). Gpt-4 omni: A comprehensive multimodal model for language, vision, and beyond. arXiv:2408.01234. https:\/\/doi.org\/10.48550\/arXiv.2408.01234","DOI":"10.48550\/arXiv.2408.01234"},{"key":"968_CR28","doi-asserted-by":"publisher","unstructured":"Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of EMNLP. https:\/\/doi.org\/10.3115\/v1\/D14-1162","DOI":"10.3115\/v1\/D14-1162"},{"key":"968_CR29","doi-asserted-by":"publisher","unstructured":"Peters, M. E.,\u00a0Neumann, M.,\u00a0Iyyer, M., et\u00a0al. (2018). Deep contextualized word representations. In Proceedings of ACL. https:\/\/doi.org\/10.18653\/v1\/N18-1202","DOI":"10.18653\/v1\/N18-1202"},{"key":"968_CR30","doi-asserted-by":"crossref","unstructured":"Raihan, N.,\u00a0Anastasopoulos, A.,\u00a0Zampieri, M. (2025). mHumanEval - a multilingual benchmark to evaluate large language models for code generation. In L.\u00a0Chiruzzo, A.\u00a0Ritter, & L.\u00a0Wang (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2025.naacl-long.570"},{"key":"968_CR31","doi-asserted-by":"crossref","unstructured":"Raihan, N.,\u00a0Goswami, D., Puspo, S. S. C., et\u00a0al. (2024). Cseprompts: A benchmark of introductory computer science prompts. In International Symposium on Methodologies for Intelligent Systems.","DOI":"10.1007\/978-3-031-62700-2_5"},{"key":"968_CR32","doi-asserted-by":"publisher","unstructured":"Raihan, N.,\u00a0Newman, C., &\u00a0Zampieri, M. (2024). Code llms: A taxonomy-based survey. In 2024 IEEE International Conference on Big Data (BigData). https:\/\/doi.org\/10.1109\/BigData62323.2024.10826108","DOI":"10.1109\/BigData62323.2024.10826108"},{"key":"968_CR33","doi-asserted-by":"publisher","unstructured":"Raihan, N.,\u00a0Santos, J., &\u00a0Zampieri, M. (2024). Mojobench: Language modeling and benchmarks for mojo. arXiv:2410.17736. https:\/\/doi.org\/10.48550\/arXiv.2410.17736 .","DOI":"10.48550\/arXiv.2410.17736"},{"key":"968_CR34","doi-asserted-by":"publisher","unstructured":"Raihan, N., Siddiq, M. L., Santos, J. C., et\u00a0al. (2025). Large language models in computer science education: A systematic literature review. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1. https:\/\/doi.org\/10.1145\/3699459.3703350","DOI":"10.1145\/3699459.3703350"},{"key":"968_CR35","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","volume":"8","author":"A Rogers","year":"2020","unstructured":"Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8, 842\u2013866. https:\/\/doi.org\/10.1162\/tacl_a_00349","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"968_CR36","doi-asserted-by":"publisher","unstructured":"Roziere, B.,\u00a0Gehring, J.,\u00a0Gloeckle, F., et\u00a0al. (2023). Code llama: Open foundation models for code. arxiv:2308.12950. https:\/\/doi.org\/10.48550\/arXiv.2308.12950","DOI":"10.48550\/arXiv.2308.12950"},{"key":"968_CR37","doi-asserted-by":"publisher","unstructured":"Savelka, J.,\u00a0Agarwal, A.,\u00a0Bogart, C., et\u00a0al. (2023). Can generative pre-trained transformers (gpt) pass assessments in higher education programming courses? arXiv:2303.09325. https:\/\/doi.org\/10.48550\/arXiv.2303.09325 .","DOI":"10.48550\/arXiv.2303.09325"},{"key":"968_CR38","doi-asserted-by":"publisher","unstructured":"Siddiq, M. L., Dristi, S. B.,\u00a0Saha, J., et\u00a0al. (2024). The fault in our stars: Quality assessment of code generation benchmarksn. In 24th IEEE International Conference on Source Code Analysis and Manipulation (SCAM). https:\/\/doi.org\/10.1109\/SCAM63248.2024.00018","DOI":"10.1109\/SCAM63248.2024.00018"},{"key":"968_CR39","doi-asserted-by":"publisher","unstructured":"Sok, S., & Heng, K. (2023). Chatgpt for education and research: A review of benefits and risks. Available at SSRN, 4378735. https:\/\/doi.org\/10.2139\/ssrn.4378735","DOI":"10.2139\/ssrn.4378735"},{"key":"968_CR40","doi-asserted-by":"publisher","unstructured":"Surameery, N. M. S., & Shakor, M. Y. (2023). Use chat gpt to solve programming bugs. International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290,\u00a03(01): 17\u201322. https:\/\/doi.org\/10.55529\/ijitc.31.17.22","DOI":"10.55529\/ijitc.31.17.22"},{"key":"968_CR41","doi-asserted-by":"publisher","unstructured":"Svyatkovskiy, A., Zhao, S. K., Fu, S., et al. (2021). Fast and memory-efficient neural code completion. In ICML. https:\/\/doi.org\/10.1109\/MSR52588.2021.00045","DOI":"10.1109\/MSR52588.2021.00045"},{"key":"968_CR42","doi-asserted-by":"publisher","unstructured":"Vaswani, A.,\u00a0Shazeer, N.,\u00a0Parmar, N., et\u00a0al. (2017). Attention is all you need. In I.\u00a0Guyon, U.\u00a0V. Luxburg, S.\u00a0Bengio, & et\u00a0al. (Eds.), Advances in Neural Information Processing Systems, vol.\u00a030. Curran Associates, Inc. https:\/\/doi.org\/10.48550\/arXiv.1706.03762","DOI":"10.48550\/arXiv.1706.03762"},{"key":"968_CR43","doi-asserted-by":"publisher","unstructured":"Wang, B., &\u00a0Komatsuzaki, A. (2021). Gpt-j-6b: A 6 billion parameter autoregressive language model. arXiv. https:\/\/doi.org\/10.5281\/zenodo.5297110. .","DOI":"10.5281\/zenodo.5297110."},{"key":"968_CR44","doi-asserted-by":"publisher","unstructured":"Wang, X.,\u00a0Wang, Y.,\u00a0Mi, F., et\u00a0al. (2021). Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation. arXiv:2108.04556. https:\/\/doi.org\/10.48550\/arXiv.2108.04556 .","DOI":"10.48550\/arXiv.2108.04556"},{"key":"968_CR45","doi-asserted-by":"publisher","unstructured":"Wang, Y.,\u00a0Wang, W.,\u00a0Joty, S., et\u00a0al. (2021). Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https:\/\/doi.org\/10.18653\/v1\/2021.emnlp-main.685","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"968_CR46","doi-asserted-by":"publisher","unstructured":"Wei, Y.,\u00a0Wang, Z.,\u00a0Liu, J., et\u00a0al. (2023). Magicoder: Source code is all you need. arXiv:2312.02120. https:\/\/doi.org\/10.48550\/arXiv.2312.02120","DOI":"10.48550\/arXiv.2312.02120"},{"key":"968_CR47","doi-asserted-by":"publisher","unstructured":"Zan, X. V.,\u00a0Deng, M.,\u00a0Yang, D., et\u00a0al. (2022). A survey of benchmarks for natural language to code generation. In ACL. https:\/\/doi.org\/10.18653\/v1\/2022.acl-long.412","DOI":"10.18653\/v1\/2022.acl-long.412"},{"key":"968_CR48","doi-asserted-by":"publisher","unstructured":"Zhang, S.J., S.\u00a0Florin, Lee, and et\u00a0al. 2023. Exploring the mit mathematics and eecs curriculum using large language models. arXiv preprint arXiv:2306.08997. https:\/\/doi.org\/10.48550\/arXiv.2306.08997 .","DOI":"10.48550\/arXiv.2306.08997"}],"container-title":["Journal of Intelligent Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10844-025-00968-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10844-025-00968-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10844-025-00968-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,16]],"date-time":"2026-02-16T07:39:55Z","timestamp":1771227595000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10844-025-00968-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,16]]},"references-count":48,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,2]]}},"alternative-id":["968"],"URL":"https:\/\/doi.org\/10.1007\/s10844-025-00968-y","relation":{},"ISSN":["0925-9902","1573-7675"],"issn-type":[{"value":"0925-9902","type":"print"},{"value":"1573-7675","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,16]]},"assertion":[{"value":"28 October 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 July 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 August 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}