{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T11:21:03Z","timestamp":1773055263406,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T00:00:00Z","timestamp":1773014400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Reliable evaluation of large language models (LLMs) for educational use requires benchmarks that reflect exam constraints, instructor grading practices, and the operational consequences of thresholded decisions. This paper introduces ExamQ-Gen, an instructor-in-the-loop benchmark that couples two tasks: (i) an LLM answering university-style exam questions and (ii) decision-support grading aligned with an instructor reference. Automatic grading is used for triage and feedback; in practice, ExamQ-Gen supports instructor-led exam authoring and provides grading recommendations, while the instructor issues the final grade and pass\/fail decision. ExamQ-Gen is constructed from the course content by using an LLM to generate exam-style questions directly from the lecture materials, producing a course-derived question set suitable for controlled experimentation. The benchmark then instantiates contrasting exam conditions, including instructor-authored (HUMAN) versus pipeline-generated (PIPELINE) artifacts, to evaluate robustness under distribution shifts that can occur when exam questions and answers are produced through different generation workflows. Using two LLM \u201cstudents\u201d (Llama3-8B-Instruct and Mistral-7B-Instruct) and an LLM-based grader, we compare automatic grading against an instructor reference on a 1\u201310 score scale and at the decision level induced by the operational pass policy (pass if score \u2265 9). Accordingly, our conclusions are conditioned on the two evaluated student models. Score-level agreement is strong under HUMAN conditions but degrades substantially under PIPELINE conditions, indicating condition-dependent stability. At the pass threshold, decision errors are highly asymmetric, with false fails dominating false passes, meaning that conservative grading may appear safe while producing credit denial. A severity-focused analysis isolates a high-stakes failure mode\u2014denial of instructor-perfect answers\u2014and shows that, in the most affected PIPELINE condition, the perfect-pass miss rate reaches 0.926 (50\/54), consistent with systematic conservatism rather than borderline noise. Overall, the results highlight that aggregate score agreement and accuracy are insufficient for instructor-controlled exam deployment and motivate reporting practices that combine disaggregated score agreement, threshold-based error asymmetry with uncertainty, and severity-aware diagnostics under exam-relevant condition shifts.<\/jats:p>","DOI":"10.3390\/computers15030177","type":"journal-article","created":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T08:58:45Z","timestamp":1773046725000},"page":"177","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["ExamQ-Gen: Instructor-in-the-Loop Generation of Self-Contained Exam Questions from Course Materials and Decision-Support Grading"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1849-3072","authenticated-orcid":false,"given":"Catalin","family":"Anghel","sequence":"first","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1155-5274","authenticated-orcid":false,"given":"Emilia","family":"Pecheanu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2537-6713","authenticated-orcid":false,"given":"Andreea Alexandra","family":"Anghel","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computer Science, Electrical and Electronic Engineering, \u201cDun\u0103rea de Jos\u201d University of Galati, 800008 Galati, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9970-2556","authenticated-orcid":false,"given":"Marian Viorel","family":"Craciun","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0935-4713","authenticated-orcid":false,"given":"Adina","family":"Cocu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Emirtekin, E. (2025). Large Language Model-Powered Automated Assessment: A Systematic Review. Appl. Sci., 15.","DOI":"10.3390\/app15105683"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Aggarwal, D., Sil, P., Raman, B., and Bhattacharyya, P. (2025, January 22\u201326). \u201cI Understand Why I Got This Grade\u201d: Automatic Short Answer Grading (ASAG) with Feedback. Proceedings of the Artificial Intelligence in Education (AIED 2025), Cham, Switzerland.","DOI":"10.1007\/978-3-031-98420-4_22"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Pan, Y., and Nehm, R.H. (2025). Large Language Model and Traditional Machine Learning Scoring of Evolutionary Explanations: Benefits and Drawbacks. Educ. Sci., 15.","DOI":"10.3390\/educsci15060676"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/s10648-023-09823-4","article-title":"Effects of Rubrics on Academic Performance, Self-Regulated Learning, and self-Efficacy: A Meta-analytic Review","volume":"35","author":"Panadero","year":"2023","journal-title":"Educ. Psychol. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1109\/TLT.2023.3330531","article-title":"Natural Language Processing of Student\u2019s Feedback to Instructors: A Systematic Review","volume":"17","author":"Sunar","year":"2023","journal-title":"IEEE Trans. Learn. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Cisneros-Gonz\u00e1lez, J., Gordo-Herrera, N., Barcia-Santos, I., and S\u00e1nchez-Soriano, J. (2025). JorGPT: Instructor-Aided Grading of Programming Assignments with Large Language Models (LLMs). Future Internet, 17.","DOI":"10.3390\/fi17060265"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"010126","DOI":"10.1103\/PhysRevPhysEducRes.21.010126","article-title":"Grading Explanations of Problem-Solving Process and Generating Feedback Using Large Language Models at Human-Level Accuracy","volume":"21","author":"Chen","year":"2025","journal-title":"Phys. Rev. Phys. Educ. Res."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Blagec, K., Dorffner, G., Moradi, M., Ott, S., and Samwald, M. (2022, January 26). A global analysis of metrics used for measuring performance in natural language processing. Proceedings of the NLP Power! The First Workshop on Efficient Benchmarking in NLP, Dublin, Ireland. Available online: https:\/\/aclanthology.org\/2022.nlppower-1.6\/.","DOI":"10.18653\/v1\/2022.nlppower-1.6"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Hackl, V., Krainz, A., and Bock, A. (2023). Is GPT-4 a Reliable Rater? Evaluating Consistency in GPT-4\u2019s Text Ratings. Front. Educ., 8.","DOI":"10.3389\/feduc.2023.1272229"},{"key":"ref_10","unstructured":"Celikyilmaz, A., Clark, E., and Gao, J. (2020). Evaluation of Text Generation: A Survey. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Anghel, C., Anghel, A.A., Pecheanu, E., Cocu, A., and Istrate, A. (2025). Diagnosing Bias and Instability in LLM Evaluation: A Scalable Pairwise Meta-Evaluator. Information, 16.","DOI":"10.3390\/info16080652"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2889","DOI":"10.1007\/s40593-025-00486-6","article-title":"The Impact of Explainable AI on Teachers\u2019 Trust and Acceptance of AI EdTech Recommendations: The Power of Domain-specific Explanations","volume":"35","author":"Cukurova","year":"2025","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Melo, E., Silva, I., Costa, D.G., Viegas, C.M.D., and Barros, T.M. (2022). On the Use of eXplainable Artificial Intelligence to Evaluate School Dropout. Educ. Sci., 12.","DOI":"10.3390\/educsci12120845"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Anghel, C., Anghel, A.A., Pecheanu, E., Susnea, I., Cocu, A., and Istrate, A. (2025). Multi-Model Dialectical Evaluation of LLM Reasoning Chains: A Structured Framework with Dual Scoring Agents. Informatics, 12.","DOI":"10.3390\/informatics12030076"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Schneider, J., Schenk, B., and Niklaus, C. (2024, January 2\u20134). Towards LLM-Based Autograding for Short Textual Answers. Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024), Angers, France.","DOI":"10.5220\/0012552200003693"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Duong, T.N.B., and Chai, Y.M. (2024, January 8\u201311). Automatic Grading of Short Answers Using Large Language Models in Software Engineering Courses. Proceedings of the 2024 IEEE Global Engineering Education Conference (EDUCON), Kos, Greece.","DOI":"10.1109\/EDUCON60312.2024.10578839"},{"key":"ref_17","unstructured":"Jukiewicz, M. (2025). A Systematic Comparison of Large Language Models for Automated Assignment Assessment in Programming Education: Exploring the Importance of Architecture and Vendor. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2592430","DOI":"10.1080\/10872981.2025.2592430","article-title":"When AI Models Take the Exam: Large Language Models vs. Medical Students on Multiple-Choice Course Exams","volume":"30","author":"Renato","year":"2025","journal-title":"Med. Educ. Online"},{"key":"ref_19","unstructured":"Gaggioli, A., Casaburi, G., Ercolani, L., Collov\u00e0, F., Torre, P., and Davide, F. (2025). Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nikolovski, V., Trajanov, D., and Chorbev, I. (2025). Advancing AI in Higher Education: A Comparative Study of Large Language Model-Based Agents for Exam Question Generation, Improvement, and Evaluation. Algorithms, 18.","DOI":"10.3390\/a18030144"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Scaria, N., Dharani Chenna, S., and Subramani, D. (2024, January 8\u201312). Automated Educational Question Generation at Different Bloom\u2019s Skill Levels Using Large Language Models: Strategies and Evaluation. Proceedings of the Artificial Intelligence in Education. AIED 2024, Palermo, Italy.","DOI":"10.1007\/978-3-031-64299-9_12"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"100298","DOI":"10.1016\/j.caeai.2024.100298","article-title":"Analysis of LLMs for Educational Question Classification and Generation","volume":"7","author":"Romadhony","year":"2024","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., and Kedzie, C. (2024, January 11\u201316). LLM-RUBRIC: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Bangkok, Thailand. Available online: https:\/\/aclanthology.org\/2024.acl-long.745.","DOI":"10.18653\/v1\/2024.acl-long.745"},{"key":"ref_24","unstructured":"Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., and Kumar, A. (2022). Holistic Evaluation of Language Models. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"44805","DOI":"10.1038\/s41598-025-28492-7","article-title":"Benchmarking ChatGPT-generated multiple-choice questions against faculty-authored items in dental education","volume":"15","author":"Kiyani","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1152\/advan.00197.2025","article-title":"Faculty versus artificial intelligence chatbot: A comparative analysis of multiple-choice question quality in physiology","volume":"49","author":"Dhanvijay","year":"2025","journal-title":"Adv. Physiol. Educ."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Arif, T., Asthana, S., and Collins-Thompson, K. (2024, January 18\u201320). Generation and Assessment of Multiple-Choice Questions from Video Transcripts using Large Language Models. Proceedings of the Eleventh ACM Conference on Learning @ Scale (L@S \u201924), Atlanta, GA, USA.","DOI":"10.1145\/3657604.3664714"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"105244","DOI":"10.1016\/j.compedu.2025.105244","article-title":"Can AI support human grading? Examining machine attention and confidence in short answer scoring","volume":"228","author":"Li","year":"2025","journal-title":"Comput. Educ."},{"key":"ref_29","unstructured":"(2025, December 11). Llama 3.3: Model Cards and Prompt Formats. Available online: https:\/\/www.llama.com\/docs\/model-cards-and-prompt-formats\/llama3_3\/."},{"key":"ref_30","unstructured":"Meta-AI (2025, December 09). Introducing Meta Llama 3. Available online: https:\/\/ai.meta.com\/blog\/meta-llama-3\/."},{"key":"ref_31","unstructured":"Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv."},{"key":"ref_32","unstructured":"(2025, December 09). Qwen\/Qwen2.5-7B-Instruct. Available online: https:\/\/huggingface.co\/Qwen\/Qwen2.5-7B-Instruct."},{"key":"ref_33","unstructured":"Neo4j, I. (2025, December 12). Neo4j Graph Database Platform. Available online: https:\/\/neo4j.com\/product\/neo4j-graph-database\/."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2020, January 16\u201320). Transformers: State-of-the-Art Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Stroudsburg, PA, USA (online). Available online: https:\/\/aclanthology.org\/2020.emnlp-demos.6\/.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_35","unstructured":"Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. (2022). 8-bit Optimizers via Block-wise Quantization. arXiv."},{"key":"ref_36","unstructured":"(2025, December 09). Ollama: Local Deployment of Large Language Models. Available online: https:\/\/ollama.com."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/3\/177\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T09:14:13Z","timestamp":1773047653000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/15\/3\/177"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,9]]},"references-count":36,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["computers15030177"],"URL":"https:\/\/doi.org\/10.3390\/computers15030177","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,9]]}}}