{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T16:31:01Z","timestamp":1783787461618,"version":"3.55.0"},"reference-count":53,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T00:00:00Z","timestamp":1741046400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The transformative capabilities of large language models (LLMs) are reshaping educational assessment and question design in higher education. This study proposes a systematic framework for leveraging LLMs to enhance question-centric tasks: aligning exam questions with course objectives, improving clarity and difficulty, and generating new items guided by learning goals. The research spans four university courses\u2014two theory-focused and two application-focused\u2014covering diverse cognitive levels according to Bloom\u2019s taxonomy. A balanced dataset ensures representation of question categories and structures. Three LLM-based agents\u2014VectorRAG, VectorGraphRAG, and a fine-tuned LLM\u2014are developed and evaluated against a meta-evaluator, supervised by human experts, to assess alignment accuracy and explanation quality. Robust analytical methods, including mixed-effects modeling, yield actionable insights for integrating generative AI into university assessment processes. Beyond exam-specific applications, this methodology provides a foundational approach for the broader adoption of AI in post-secondary education, emphasizing fairness, contextual relevance, and collaboration. The findings offer a comprehensive framework for aligning AI-generated content with learning objectives, detailing effective integration strategies, and addressing challenges such as bias and contextual limitations. Overall, this work underscores the potential of generative AI to enhance educational assessment while identifying pathways for responsible implementation.<\/jats:p>","DOI":"10.3390\/a18030144","type":"journal-article","created":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T13:31:31Z","timestamp":1741095091000},"page":"144","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Advancing AI in Higher Education: A Comparative Study of Large Language Model-Based Agents for Exam Question Generation, Improvement, and Evaluation"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2207-1526","authenticated-orcid":false,"given":"Vlatko","family":"Nikolovski","sequence":"first","affiliation":[{"name":"Faculty of Computer Science and Engineering, \u201cSs. Cyril and Methodius\u201d University in Skopje, 1000 Skopje, North Macedonia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3105-6010","authenticated-orcid":false,"given":"Dimitar","family":"Trajanov","sequence":"additional","affiliation":[{"name":"Faculty of Computer Science and Engineering, \u201cSs. Cyril and Methodius\u201d University in Skopje, 1000 Skopje, North Macedonia"},{"name":"Department of Computer Science, Metropolitan College, Boston University, Boston, MA 02215, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9648-4438","authenticated-orcid":false,"given":"Ivan","family":"Chorbev","sequence":"additional","affiliation":[{"name":"Faculty of Computer Science and Engineering, \u201cSs. Cyril and Methodius\u201d University in Skopje, 1000 Skopje, North Macedonia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,4]]},"reference":[{"key":"ref_1","unstructured":"Wang, S., Xu, T., Li, H., Zhang, C., Liang, J., Tang, J., Yu, P.S., and Wen, Q. (2024). Large language models for education: A survey and outlook. arXiv."},{"key":"ref_2","unstructured":"Pel\u00e1ez-S\u00e1nchez, I.C., Velarde-Camaqui, D., and Glasserman-Morales, L.D. (2023). The impact of large language models on higher education. Front. Educ., 8."},{"key":"ref_3","unstructured":"Baierl, J.D. (2023). Applications of Large Language Models in Education: Literature Review and Implications. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"75264","DOI":"10.1109\/ACCESS.2020.2988510","article-title":"Artificial intelligence in education: A review","volume":"8","author":"Chen","year":"2020","journal-title":"IEEE Access"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s40593-019-00186-y","article-title":"A systematic review of automatic question generation for educational purposes","volume":"30","author":"Kurdi","year":"2020","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Scaria, N., Dharani Chenna, S., and Subramani, D. (2024). Automated Educational Question Generation at Different Bloom\u2019s Skill Levels Using Large Language Models: Strategies and Evaluation. Artificial Intelligence in Education, Springer.","DOI":"10.1007\/978-3-031-64299-9_12"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bulathwela, S., Muse, H., and Yilmaz, E. (2023). Scalable educational question generation with pre-trained language models. Artificial Intelligence in Education, Springer.","DOI":"10.1007\/978-3-031-36272-9_27"},{"key":"ref_8","unstructured":"Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H., and Krathwohl, D.R. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive Domain, David McKay Company."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1186\/s41239-023-00436-z","article-title":"A meta systematic review of artificial intelligence in higher education: A call for increased ethics, collaboration, and rigour","volume":"21","author":"Bond","year":"2024","journal-title":"Int. J. Educ. Technol. High. Educ."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1007\/s11165-024-10176-3","article-title":"Exploring the impact of artificial intelligence in teaching and learning of science: A systematic review of empirical research","volume":"54","author":"Almasri","year":"2024","journal-title":"Res. Sci. Educ."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"80174","DOI":"10.1109\/ACCESS.2022.3194063","article-title":"Knowledge graphs in education and employability: A survey on applications and techniques","volume":"10","author":"Fettach","year":"2022","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Bulathwela, S., P\u00e9rez-Ortiz, M., Yilmaz, E., and Shawe-Taylor, J. (2022). Power to the learner: Towards human-intuitive and integrative recommendations with open educational resources. Sustainability, 14.","DOI":"10.3390\/su141811682"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Alotaibi, N.S. (2024). The Impact of AI and LMS Integration on the Future of Higher Education: Opportunities, Challenges, and Strategies for Transformation. Sustainability, 16.","DOI":"10.3390\/su162310357"},{"key":"ref_14","unstructured":"Pamboris, A. (2024, December 27). The Impact of LLMs on Higher Education: Balancing Innovation with Ethics, Integrity, and Fairness. Eloquence AI, Available online: https:\/\/eloquenceai.eu\/the-impact-of-llms-on-higher-education-balancing-innovation-with-ethics-integrity-and-fairness\/."},{"key":"ref_15","unstructured":"Bouchard, D. (2024). An actionable framework for assessing bias and fairness in large language model use cases. arXiv, Available online: https:\/\/arxiv.org\/abs\/2407.10853."},{"key":"ref_16","unstructured":"University, A.S. (2025, January 13). Evaluation Framework Sets a New Benchmark for Ethical AI. ASU News, Available online: https:\/\/tech.asu.edu\/features\/evaluation-framework-sets-new-benchmark-ethical-ai."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"11483","DOI":"10.1007\/s10639-023-12249-8","article-title":"Few-shot is enough: Exploring ChatGPT prompt engineering method for automatic question generation in english education","volume":"29","author":"Lee","year":"2024","journal-title":"Educ. Inf. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D.H., and MacNeil, S. (2023, January 18\u201321). Generating multiple choice questions for computing courses using large language models. Proceedings of the 2023 IEEE Frontiers in Education Conference (FIE), College Station, TX, USA.","DOI":"10.1109\/FIE58773.2023.10342898"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Duong-Trung, N., Wang, X., and Krav\u010d\u00edk, M. (2024). BloomLLM: Large Language Models Based Question Generation Combining Supervised Fine-Tuning and Bloom\u2019s Taxonomy. Technology Enhanced Learning for Inclusive and Equitable Quality Education, Springer.","DOI":"10.1007\/978-3-031-72312-4_11"},{"key":"ref_20","first-page":"23164","article-title":"CyberQ: Generating Questions and Answers for Cybersecurity Education Using Knowledge Graph-Augmented LLMs","volume":"38","author":"Agrawal","year":"2024","journal-title":"AAAI Conf. Artif. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Bhowmick, A.K., Jagmohan, A., Vempaty, A., Dey, P., Hall, L., Hartman, J., Kokku, R., and Maheshwari, H. (2023). Automating question generation from educational text. Artificial Intelligence XL, Springer.","DOI":"10.1007\/978-3-031-47994-6_38"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lodovico Molina, I., \u0160v\u00e1bensk\u1ef3, V., Minematsu, T., Chen, L., Okubo, F., and Shimada, A. (2024). Comparison of Large Language Models for Generating Contextually Relevant Questions. Technology Enhanced Learning for Inclusive and Equitable Quality Education, Springer.","DOI":"10.1007\/978-3-031-72312-4_18"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gonzalez, H., Dugan, L., Miltsakaki, E., Cui, Z., Ren, J., Li, B., Upadhyay, S., Ginsberg, E., and Callison-Burch, C. (2023, January 13). Enhancing human summaries for question-answer generation in education. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Toronto, ON, Canada.","DOI":"10.18653\/v1\/2023.bea-1.9"},{"key":"ref_24","unstructured":"Lee, J., Smith, D., Woodhead, S., and Lan, A. (2024). Math Multiple Choice Question Generation via Human-Large Language Model Collaboration. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Shin, J., Guo, Q., and Gierl, M.J. (2019). Multiple-choice item distractor development using topic modeling approaches. Front. Psychol., 10.","DOI":"10.3389\/fpsyg.2019.00825"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Stasaski, K., and Hearst, M.A. (2017, January 8). Multiple choice question generation utilizing an ontology. Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, Copenhagen, Denmark.","DOI":"10.18653\/v1\/W17-5034"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Kwan, C.C.L. (2023). Exploring ChatGPT-Generated assessment scripts of probability and engineering statistics from bloom\u2019s taxonomy. Technology Enhanced Learning for Inclusive and Equitable Quality Education, Springer.","DOI":"10.1007\/978-981-99-8255-4_24"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"100252","DOI":"10.1016\/j.caeai.2024.100252","article-title":"Automatic question-answer pairs generation using pre-trained large language models in higher education","volume":"6","author":"Ling","year":"2024","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Elkins, S., Kochmar, E., Serban, I., and Cheung, J.C. (2023, January 3\u20137). How useful are educational questions generated by large language models?. Proceedings of the International Conference on Artificial Intelligence in Education, Tokyo, Japan.","DOI":"10.1007\/978-3-031-36336-8_83"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Maity, S., and Deroy, A. (2024). The future of learning in the age of generative ai: Automated question generation and assessment with large language models. arXiv.","DOI":"10.35542\/osf.io\/ncjek"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Scarlatos, A., Smith, D., Woodhead, S., and Lan, A. (2024, January 8\u201312). Improving the validity of automatically generated feedback via reinforcement learning. Proceedings of the International Conference on Artificial Intelligence in Education, Recife, Brazil.","DOI":"10.1007\/978-3-031-64302-6_20"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.","DOI":"10.18653\/v1\/D16-1264"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1007\/s40593-018-0168-1","article-title":"Personalizing algebra to students\u2019 individual interests in an intelligent tutoring system: Moderators of impact","volume":"29","author":"Walkington","year":"2019","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"ref_34","unstructured":"Xie, W., Niu, J., Xue, C.J., and Guan, N. (2024). Grade Like a Human: Rethinking Automated Assessment with Large Language Models. arXiv."},{"key":"ref_35","unstructured":"Ishida, T., Liu, T., Wang, H., and Cheung, W.K. (2024). Large Language Models as Partners in Student Essay Evaluation. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Elkins, S., Kochmar, E., Cheung, J.C., and Serban, I. (2024). How Teachers Can Use Large Language Models and Bloom\u2019s Taxonomy to Create Educational Quizzes. arXiv.","DOI":"10.1609\/aaai.v38i21.30353"},{"key":"ref_37","unstructured":"Diagrams.net (2024, December 27). Online Diagram Software and Flowchart Maker. Available online: https:\/\/www.diagrams.net\/."},{"key":"ref_38","unstructured":"\u0141ukawski, K. (2024, December 27). Hybrid Search Revamped\u2014Building with Qdrant\u2019s Query API. Qdrant Articles, Available online: https:\/\/qdrant.tech\/articles\/hybrid-search\/."},{"key":"ref_39","first-page":"9459","article-title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_40","unstructured":"Peng, B., Zhu, Y., Liu, Y., Bo, X., Shi, H., Hong, C., Zhang, Y., and Tang, S. (2024). Graph Retrieval-Augmented Generation: A Survey. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Bui, T., Tran, O., Nguyen, P., Ho, B., Nguyen, L., Bui, T., and Quan, T. (2024). Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUT. arXiv.","DOI":"10.1145\/3643479.3662055"},{"key":"ref_42","unstructured":"Google (2025, January 13). Introducing Gemini 2.0: Our New AI Model for the Agentic Era. Available online: https:\/\/blog.google\/technology\/google-deepmind\/google-gemini-ai-update-december-2024\/."},{"key":"ref_43","unstructured":"Qdrant (2025, January 13). Vector Search Engine and Database. Available online: https:\/\/qdrant.tech\/."},{"key":"ref_44","unstructured":"Neo4j, Inc. (2023). Neo4j Graph Database Platform, Neo4j, Inc.. Available online: https:\/\/neo4j.com\/."},{"key":"ref_45","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (2024). The Llama 3 Herd of Models. arXiv."},{"key":"ref_46","unstructured":"Nikolovski, V. (2025, February 10). Meta-Llama-3.1-8B-Instruct-finki-edu-5c. Available online: https:\/\/huggingface.co\/vlada22\/Meta-Llama-3.1-8B-Instruct-finki-edu-5c."},{"key":"ref_47","unstructured":"Nikolovski, V. (2025, January 13). Meta-Llama-3.1-8B-Instruct-finki-edu-4courses. Available online: https:\/\/huggingface.co\/vlada22\/Meta-Llama-3.1-8B-Instruct-finki-edu-4courses."},{"key":"ref_48","unstructured":"OpenAI (2025, January 13). Introducing GPT-4o and More Tools to ChatGPT Free Users. Available online: https:\/\/openai.com\/index\/gpt-4o-and-more-tools-to-chatgpt-free\/."},{"key":"ref_49","unstructured":"Pandas Development Team (2025, January 13). Pandas: Python Data Analysis Library. Available online: https:\/\/pandas.pydata.org\/."},{"key":"ref_50","unstructured":"Seabold, S., and Perktold, J. (2025, January 13). Statsmodels: Statistical Models in Python. Available online: https:\/\/www.statsmodels.org\/."},{"key":"ref_51","unstructured":"LangChain (2023). Building Applications with LLMs Through Composability, LangChain. Available online: https:\/\/python.langchain.com\/."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., and Huang, F. (2024). mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. arXiv.","DOI":"10.18653\/v1\/2024.emnlp-industry.103"},{"key":"ref_53","unstructured":"Vlada22 (2025, February 10). Advancing AI in Higher Education. Available online: https:\/\/github.com\/vlada22\/advancing-ai-in-higher-education."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/3\/144\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:47:13Z","timestamp":1760028433000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/3\/144"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,4]]},"references-count":53,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["a18030144"],"URL":"https:\/\/doi.org\/10.3390\/a18030144","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,4]]}}}