{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T09:05:16Z","timestamp":1783933516482,"version":"3.55.0"},"reference-count":64,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T00:00:00Z","timestamp":1783900800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Science, Technological Development and Innovation of the Republic of Serbia","award":["451-03-137\/2025-03\/200103, 451-03-47\/2025-01\/200104"],"award-info":[{"award-number":["451-03-137\/2025-03\/200103, 451-03-47\/2025-01\/200104"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>The time complexity of algorithms is a critical concept in computer science and engineering, and it is recognized as a fundamental topic in the Association for Computing Machinery (ACM) curricular recommendations. In alignment with these guidelines, numerous universities worldwide incorporate this topic into their introductory computer science courses, which typically have large student enrollments annually. Consequently, there is an increasing demand for automation in both instructional and assessment processes. This study explores the potential of Large Language Models (LLMs) to assist teaching staff in generating source code segments with predefined time complexity and determining the time complexity of given code segments, with applications in educational and examination contexts. We proposed a novel methodology for LLM evaluation in the aforementioned context and evaluated three prominent LLMs: ChatGPT, Gemini, and Llama, on their ability to generate and analyze C code segments exhibiting linear, logarithmic, quadratic, and exponential time complexities. A framework was developed to automate the prompt and segment generation and time complexity determination using two mainstream prompt engineering methods: zero-shot and chain-of-thought, and assessed the differences in code generation and time complexity analysis. A total of 960 generated segments were assessed on the correctness of time complexity, structural appropriateness, and suitability for exam use. The results suggest that ChatGPT is the most suitable LLM for generating segments with predefined time complexity (success rate goes up to 61%). All LLMs yielded the best results in generating linear segments, while exponential complexity posed the greatest challenge overall. A subset of generated segments was extracted to evaluate the time complexity determination capabilities. All three LLMs were asked to find the time complexity of each extracted segment. The most accurate LLM is ChatGPT (79.6%). We also assessed how good each LLM is in determining the time complexity of segments generated by itself. Llama outperforms others in that task (83% of successful determinations) when the zero-shot prompt method is used. The findings suggest that current LLMs cannot fully automate question generation and time complexity problem solving. However, they can substantially support the process and reduce the workload for educators.<\/jats:p>","DOI":"10.7717\/peerj-cs.3946","type":"journal-article","created":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T08:12:26Z","timestamp":1783930346000},"page":"e3946","source":"Crossref","is-referenced-by-count":0,"title":["Assessing the effectiveness of large language models for generating and estimating time complexity of code segments"],"prefix":"10.7717","volume":"12","author":[{"given":"\u0110or\u0111e","family":"Pe\u0161i\u0107","sequence":"first","affiliation":[{"name":"School of Electrical Engineering, University of Belgrade, Belgrade, Serbia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5396-0644","authenticated-orcid":true,"given":"Milena","family":"Vujo\u0161evi\u0107 Jani\u010di\u0107","sequence":"additional","affiliation":[{"name":"Faculty of Mathematics, University of Belgrade, Belgrade, Serbia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7369-4010","authenticated-orcid":true,"given":"Marko","family":"Mi\u0161i\u0107","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering, University of Belgrade, Belgrade, Serbia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jelica","family":"Proti\u0107","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering, University of Belgrade, Belgrade, Serbia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4443","published-online":{"date-parts":[[2026,7,13]]},"reference":[{"key":"10.7717\/peerj-cs.3946\/ref-1","doi-asserted-by":"publisher","DOI":"10.1145\/3173161","volume-title":"Information technology curricula 2017: curriculum guidelines for baccalaureate degree programs in information technology","author":"ACM","year":"2017"},{"key":"10.7717\/peerj-cs.3946\/ref-2","article-title":"Curricula recommendations","author":"ACM","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-3","volume-title":"Computer engineering curricula 2016: curriculum guidelines for undergraduate degree programs in computer engineering","author":"ACM\/IEEE","year":"2016"},{"key":"10.7717\/peerj-cs.3946\/ref-4","doi-asserted-by":"publisher","DOI":"10.1145\/3664191","volume-title":"Computer science curricula 2023","author":"ACM\/IEEE","year":"2024"},{"key":"10.7717\/peerj-cs.3946\/ref-5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.01687","article-title":"\u201cWhich LLM should I use?\u201d: evaluating LLMs for tasks performed by undergraduate computer science students","author":"Agarwal","year":"2024"},{"issue":"2","key":"10.7717\/peerj-cs.3946\/ref-6","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1080\/08993400500150747","article-title":"A survey of automated assessment approaches for programming assignments","volume":"15","author":"Ala-Mutka","year":"2005","journal-title":"Computer Science Education"},{"issue":"10","key":"10.7717\/peerj-cs.3946\/ref-7","doi-asserted-by":"publisher","first-page":"45","DOI":"10.14569\/ijacsa.2022.0131006","article-title":"A review of automatic question generation in teaching programming","volume":"13","author":"Alshboul","year":"2022","journal-title":"International Journal of Advanced Computer Science and Applications"},{"key":"10.7717\/peerj-cs.3946\/ref-8","article-title":"Claude","author":"Anthropic","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-9","article-title":"Large language models: a new frontier in artificial intelligence","author":"Aragon","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.08679","article-title":"Chain-of-thought reasoning in the wild is not always faithful","author":"Arcuschin","year":"2025"},{"issue":"11","key":"10.7717\/peerj-cs.3946\/ref-11","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1109\/MC.2015.345","article-title":"SE 2014: curriculum guidelines for undergraduate degree programs in software engineering","volume":"48","author":"Ardis","year":"2015","journal-title":"Computer"},{"issue":"4","key":"10.7717\/peerj-cs.3946\/ref-12","first-page":"1058","article-title":"Automating the knowledge assessment workflow for large student groups: a development experience","volume":"31","author":"Bo\u0161njakovi\u0107","year":"2015","journal-title":"The International Journal of Engineering Education"},{"issue":"2","key":"10.7717\/peerj-cs.3946\/ref-13","doi-asserted-by":"publisher","first-page":"e1845","DOI":"10.7717\/peerj-cs.1845","article-title":"An integrative decision-making framework to guide policies on regulating ChatGPT usage","volume":"10","author":"Bukar","year":"2024","journal-title":"PeerJ Computer Science"},{"key":"10.7717\/peerj-cs.3946\/ref-14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.04477","article-title":"A comparative study of code generation using ChatGPT 3.5 across 10 programming languages","author":"Buscemi","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.15242","article-title":"BigO (Bench)\u2013can LLMS generate code with controlled time and space complexity?","author":"Chambon","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-16","first-page":"1573","article-title":"Parameterized and automated assessment on an introductory programming course","author":"de Assis Zampirolli","year":"2020"},{"key":"10.7717\/peerj-cs.3946\/ref-17","article-title":"AI","author":"Deepseek","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-18","article-title":"E-Lab: web-based system for automatic assessment of programming problems","author":"Delev","year":"2012"},{"key":"10.7717\/peerj-cs.3946\/ref-19","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/s0066-4138(63)80015-4","article-title":"ALGOL 60 translation: an ALGOL 60 translator for the X1 and making a translator for ALGOL 60","volume":"1","author":"Dijkstra","year":"1961","journal-title":"ALGOL Bulletin"},{"issue":"5","key":"10.7717\/peerj-cs.3946\/ref-20","doi-asserted-by":"publisher","first-page":"775","DOI":"10.1002\/cae.21750","article-title":"Transition from traditional to LMS supported examining: a case study in computer engineering","volume":"24","author":"Draskovic","year":"2016","journal-title":"Computer Applications in Engineering Education"},{"key":"10.7717\/peerj-cs.3946\/ref-21","author":"Dujmovi\u0107","year":"2004","journal-title":"Programming languages and programming methods\u2014selected chapters"},{"key":"10.7717\/peerj-cs.3946\/ref-22","article-title":"On difficult topics in theoretical computer science education","author":"Enstr\u00f6m","year":"2014"},{"key":"10.7717\/peerj-cs.3946\/ref-23","article-title":"Prompt engineering techniques","author":"Gadesha","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-24","article-title":"Copilot","author":"GitHub","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-25","article-title":"Introducing Gemini: our largest and most capable AI model","author":"Google","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.02314","article-title":"Researchcodebench: benchmarking LLMS on implementing novel machine learning research code","author":"Hua","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-27","first-page":"569","article-title":"Improved program repair methods using refactoring with GPT models","volume":"1","author":"Ishizue","year":"2024"},{"key":"10.7717\/peerj-cs.3946\/ref-28","first-page":"303","article-title":"Gemini-the most powerful LLM: myth or truth","author":"Islam","year":"2024"},{"key":"10.7717\/peerj-cs.3946\/ref-29","first-page":"77","article-title":"Evaluating LLM-generated worked examples in an introductory programming course","author":"Jury","year":"2024"},{"key":"10.7717\/peerj-cs.3946\/ref-30","first-page":"269","article-title":"The analysis of algorithms","volume":"3","author":"Knuth","year":"1970"},{"key":"10.7717\/peerj-cs.3946\/ref-31","first-page":"624","article-title":"Evaluating language models for generating and judging programming feedback","volume":"1","author":"Koutcheme","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-32","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.02870","article-title":"Loki\u2019s dance of illusions: a comprehensive survey of hallucination in large language models","author":"Li","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.13851","article-title":"Which is a better programming assistant? A comparative study between ChatGPT and stack overflow","author":"Liu","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-34","first-page":"647","article-title":"Llama-reviewer: advancing code review automation with large language models through parameter-efficient fine-tuning","author":"Lu","year":"2023"},{"issue":"4","key":"10.7717\/peerj-cs.3946\/ref-35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3731756","article-title":"What should we engineer in prompts? Training humans in requirement-driven LLM use","volume":"32","author":"Ma","year":"2025","journal-title":"ACM Transactions on Computer-Human Interaction"},{"key":"10.7717\/peerj-cs.3946\/ref-36","first-page":"387","article-title":"Prompt engineering in large language models","author":"Marvin","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-37","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.07142","article-title":"Prompting science report 2: the decreasing value of chain of thought in prompting","author":"Meincke","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-38","article-title":"Llama","author":"Meta","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-39","article-title":"Copilot","author":"Microsoft","year":"2025"},{"issue":"1","key":"10.7717\/peerj-cs.3946\/ref-40","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1186\/s40537-024-01019-z","article-title":"An assessment of large language models for OpenMP-based code parallelization: a user perspective","volume":"11","author":"Mi\u0161i\u0107","year":"2024","journal-title":"Journal of Big Data"},{"key":"10.7717\/peerj-cs.3946\/ref-41","first-page":"151","article-title":"Effectiveness of ChatGPT for static analysis: how far are we?","author":"Mohajer","year":"2024"},{"issue":"1","key":"10.7717\/peerj-cs.3946\/ref-42","doi-asserted-by":"publisher","first-page":"e2105","DOI":"10.7717\/peerj-cs.2105","article-title":"Generative AI and future education: a review, theoretical validation, and authors\u2019 perspective on challenges and solutions","volume":"10","author":"Monib","year":"2024","journal-title":"PeerJ Computer Science"},{"key":"10.7717\/peerj-cs.3946\/ref-43","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.05379","article-title":"Tasty: a transformer based approach to space and time complexity","author":"Moudgalya","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-44","doi-asserted-by":"crossref","DOI":"10.1145\/3597503.3639187","article-title":"Using an LLM to help with code understanding","author":"Nam","year":"2024"},{"key":"10.7717\/peerj-cs.3946\/ref-45","article-title":"ChatGPT","author":"OpenAI","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-46","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.15601977","article-title":"djpesic\/LLMTimeComplexityComparison: 1.0.0","author":"Pe\u0161i\u0107","year":"2025","journal-title":"Zenodo"},{"key":"10.7717\/peerj-cs.3946\/ref-47","article-title":"Assembling the source code segments with the specified time complexity, using artificial intelligence tools","author":"Pe\u0161i\u0107","year":"2024a"},{"issue":"3","key":"10.7717\/peerj-cs.3946\/ref-48","doi-asserted-by":"publisher","first-page":"781","DOI":"10.2298\/csis230730015p","article-title":"A novel approach to source code assembling in the field of algorithmic complexity","volume":"21","author":"Pe\u0161i\u0107","year":"2024b","journal-title":"Computer Science and Information Systems"},{"key":"10.7717\/peerj-cs.3946\/ref-49","first-page":"F3A","article-title":"Test: tools for evaluation of students\u2019 tests-a development experience","volume":"2","author":"Protic","year":"2001"},{"key":"10.7717\/peerj-cs.3946\/ref-50","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.06971","article-title":"Chain-of-code collapse: reasoning failures in LLMs via adversarial prompting in code generation","author":"Roh","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-51","first-page":"27","article-title":"Automatic generation of programming exercises and code explanations using large language models","volume":"1","author":"Sarsa","year":"2022"},{"key":"10.7717\/peerj-cs.3946\/ref-52","first-page":"56","article-title":"Zero-shot prompting for code complexity prediction using GitHub Copilot","author":"Siddiq","year":"2023"},{"key":"10.7717\/peerj-cs.3946\/ref-53","first-page":"313","article-title":"Learning based methods for code runtime complexity prediction","volume":"42","author":"Sikka","year":"2020"},{"issue":"1","key":"10.7717\/peerj-cs.3946\/ref-54","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1112\/plms\/s2-42.1.230","article-title":"On computable numbers, with an application to the entscheidungsproblem","volume":"s2\u201342","author":"Turing","year":"1937","journal-title":"Proceedings of the London Mathematical Society"},{"key":"10.7717\/peerj-cs.3946\/ref-55","article-title":"Programming 2 course website","author":"University of Belgrade School of Electrical Engineering","year":"2024"},{"issue":"2","key":"10.7717\/peerj-cs.3946\/ref-56","doi-asserted-by":"publisher","first-page":"62","DOI":"10.58245\/ipsi.tir.2502.07","article-title":"Web application for large language model-based diagnostic analysis of correlation maps","volume":"20","author":"Vagac","year":"2025","journal-title":"IPSI Transactions on Internet Research"},{"issue":"1","key":"10.7717\/peerj-cs.3946\/ref-57","doi-asserted-by":"publisher","first-page":"205","DOI":"10.2298\/csis181220019v","article-title":"Regression verification for automated evaluation of students\u2019 programs","volume":"17","author":"Vujo\u0161evi\u0107 Jani\u010di\u0107","year":"2020","journal-title":"Computer Science and Information Systems"},{"issue":"6","key":"10.7717\/peerj-cs.3946\/ref-58","doi-asserted-by":"publisher","first-page":"1004","DOI":"10.1016\/j.infsof.2012.12.005","article-title":"Software verification and graph similarity for automated evaluation of students\u2019 assignments","volume":"55","author":"Vujo\u0161evi\u0107 Jani\u010di\u0107","year":"2013","journal-title":"Information and Software Technology"},{"key":"10.7717\/peerj-cs.3946\/ref-59","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.14256","article-title":"Impact of code context and prompting strategies on automated unit test generation with modern general-purpose large language models","author":"Walczak","year":"2025"},{"key":"10.7717\/peerj-cs.3946\/ref-60","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.05437","article-title":"Enhancing computer programming education with LLMs: a study on effective prompt engineering for Python code generation","author":"Wang","year":"2024"},{"key":"10.7717\/peerj-cs.3946\/ref-61","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.52202\/068431-1800","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"10.7717\/peerj-cs.3946\/ref-62","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1002\/cae.20260","article-title":"A genetic algorithm for generating test from a question bank","volume":"18","author":"Yildirim","year":"2010","journal-title":"Computer Applications in Engineering Education"},{"issue":"4","key":"10.7717\/peerj-cs.3946\/ref-63","doi-asserted-by":"publisher","first-page":"104159","DOI":"10.1016\/j.ipm.2025.104159","article-title":"Enhancing scientific table understanding with type-guided chain-of-thought","volume":"62","author":"Yin","year":"2025","journal-title":"Information Processing & Management"},{"issue":"5","key":"10.7717\/peerj-cs.3946\/ref-64","doi-asserted-by":"publisher","first-page":"1284","DOI":"10.1002\/cae.22385","article-title":"An experience of automated assessment in a large-scale introduction programming course","volume":"29","author":"Zampirolli","year":"2021","journal-title":"Computer Applications in Engineering Education"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-3946.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-3946.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-3946.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-3946.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T08:12:32Z","timestamp":1783930352000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-3946"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,13]]},"references-count":64,"alternative-id":["10.7717\/peerj-cs.3946"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.3946","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,13]]},"article-number":"e3946"}}