{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:57:51Z","timestamp":1783385871638,"version":"3.54.6"},"reference-count":33,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:00:00Z","timestamp":1760140800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Background and objectives: Large language models (LLMs) show promise in automating open-ended evaluation tasks, yet their reliability in rubric-based assessment remains uncertain. Variability in scoring, feedback, and rubric adherence raises concerns about transparency and pedagogical validity in educational contexts. This study introduces CourseEvalAI, a framework designed to enhance consistency and fidelity in rubric-guided evaluation by fine-tuning a general-purpose LLM with authentic university-level instructional content. Methods: The framework employs supervised fine-tuning with Low-Rank Adaptation (LoRA) on rubric-annotated answers and explanations drawn from undergraduate computer science exams. Responses generated by both the base and fine-tuned models were independently evaluated by two human raters and two LLM judges, applying dual-layer rubrics for answers (technical or argumentative) and explanations. Inter-rater reliability was reported as intraclass correlation coefficient (ICC(2,1)), Krippendorff\u2019s \u03b1, and quadratic-weighted Cohen\u2019s \u03ba (QWK), and statistical analyses included Welch\u2019s t tests with Holm\u2013Bonferroni correction, Hedges\u2019 g with bootstrap confidence intervals, and Levene\u2019s tests. All responses, scores, feedback, and metadata were stored in a Neo4j graph database for structured exploration. Results: The fine-tuned model consistently outperformed the base version across all rubric dimensions, achieving higher scores for both answers and explanations. After multiple-testing correction, only the Generative Pre-trained Transformer (GPT-4)\u2014judged Technical Answer contrast remains statistically significant; other contrasts show positive trends without passing the adjusted threshold, and no additional significance is claimed for explanation-level results. Variance in scoring decreased, inter-model agreement increased, and evaluator feedback for fine-tuned outputs contained fewer vague or critical remarks, indicating stronger rubric alignment and greater pedagogical coherence. Inter-rater reliability analyses indicated moderate human\u2013human agreement and weaker alignment of LLM judges to the human mean. Originality: CourseEvalAI integrates rubric-guided fine-tuning, dual-layer evaluation, and graph-based storage into a unified framework. This combination provides a replicable and interpretable methodology that enhances the consistency, transparency, and pedagogical value of LLM-based evaluators in higher education and beyond.<\/jats:p>","DOI":"10.3390\/computers14100431","type":"journal-article","created":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T07:17:52Z","timestamp":1760512672000},"page":"431","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["CourseEvalAI: Rubric-Guided Framework for Transparent and Consistent Evaluation of Large Language Models"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1849-3072","authenticated-orcid":false,"given":"Catalin","family":"Anghel","sequence":"first","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Gala\u021bi, \u0218tiin\u021bei St. 2, 800146 Gala\u021bi, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9970-2556","authenticated-orcid":false,"given":"Marian Viorel","family":"Craciun","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Gala\u021bi, \u0218tiin\u021bei St. 2, 800146 Gala\u021bi, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1155-5274","authenticated-orcid":false,"given":"Emilia","family":"Pecheanu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Gala\u021bi, \u0218tiin\u021bei St. 2, 800146 Gala\u021bi, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0935-4713","authenticated-orcid":false,"given":"Adina","family":"Cocu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Gala\u021bi, \u0218tiin\u021bei St. 2, 800146 Gala\u021bi, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2537-6713","authenticated-orcid":false,"given":"Andreea Alexandra","family":"Anghel","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computer Science, Electrical and Electronic Engineering, \u201cDun\u0103rea de Jos\u201d University of Gala\u021bi, 800008 Gala\u021bi, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7539-4159","authenticated-orcid":false,"given":"Paul","family":"Iacobescu","sequence":"additional","affiliation":[{"name":"Doctoral School, \u201cDun\u0103rea de Jos\u201d University of Gala\u021bi, 800201 Gala\u021bi, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Calina","family":"Maier","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, The \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Panait Sirbu Obstetrics and Gynaecology Hospital Bucharest, 060251 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Constantin Adrian","family":"Andrei","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, The \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Department of Orthopaedics, \u201cFoisor\u201d Clinical Hospital of Orthopaedics, Traumatology and Osteoarticular TB, 021382 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7676-6393","authenticated-orcid":false,"given":"Cristian","family":"Scheau","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, The \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Department of Radiology and Medical Imaging, \u201cFoisor\u201d Clinical Hospital of Orthopaedics, Traumatology and Osteoarticular TB, 021382 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7403-1684","authenticated-orcid":false,"given":"Serban","family":"Dragosloveanu","sequence":"additional","affiliation":[{"name":"Faculty of Medicine, The \u201cCarol Davila\u201d University of Medicine and Pharmacy, 050474 Bucharest, Romania"},{"name":"Department of Orthopaedics, \u201cFoisor\u201d Clinical Hospital of Orthopaedics, Traumatology and Osteoarticular TB, 021382 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Emirtekin, E. (2025). Large Language Model-Powered Automated Assessment: A Systematic Review. Appl. Sci., 15.","DOI":"10.3390\/app15105683"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Mendon\u00e7a, P.C., Quintal, F., and Mendon\u00e7a, F. (2025). Evaluating LLMs for Automated Scoring in Formative Assessments. Appl. Sci., 15.","DOI":"10.3390\/app15052787"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Pan, Y., and Nehm, R.H. (2025). Large Language Model and Traditional Machine Learning Scoring of Evolutionary Explanations: Benefits and Drawbacks. Educ. Sci., 15.","DOI":"10.3390\/educsci15060676"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Anghel, C., Anghel, A.A., Pecheanu, E., Cocu, A., and Istrate, A. (2025). Diagnosing Bias and Instability in LLM Evaluation: A Scalable Pairwise Meta-Evaluator. Information, 16.","DOI":"10.3390\/info16080652"},{"key":"ref_5","unstructured":"OpenAI (2025, July 30). GPT-4 Technical Report. Available online: https:\/\/cdn.openai.com\/papers\/gpt-4.pdf."},{"key":"ref_6","unstructured":"Meta (2025, September 17). Introducing Meta Llama 3. Available online: https:\/\/ai.meta.com\/blog\/meta-llama-3\/."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Faseeh, M., Jaleel, A., Iqbal, N., Ghani, A., Abdusalomov, A., Mehmood, A., and Cho, Y.-I. (2024). Hybrid Approach to Automated Essay Scoring: Integrating Deep Learning Embeddings with Handcrafted Linguistic Features for Improved Accuracy. Mathematics, 12.","DOI":"10.3390\/math12213416"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Doumanas, D., Soularidis, A., Spiliotopoulos, D., Vassilakis, C., and Kotis, K. (2025). Fine-Tuning Large Language Models for Ontology Engineering: A Comparative Analysis of GPT-4 and Mistral. Appl. Sci., 15.","DOI":"10.3390\/app15042146"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Jim\u00e9nez B\u00e1ez, M.V., Guti\u00e9rrez De la Cruz, M.E., Ch\u00e1vez Hern\u00e1ndez, M.M., Mart\u00ednez Castro, L.R., and Alcocer N\u00fa\u00f1ez, F.J. (2022). Digital Quality Resources Resulting from Standardized Program for Rubric Training in Medical Residents. Healthcare, 10.","DOI":"10.3390\/healthcare10112209"},{"key":"ref_10","unstructured":"Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., and Xing, E.P. (2023, January 10\u201316). Judging LLM-as-a-judge with MT-bench and Chatbot Arena. Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_11","unstructured":"Thakur, A.S., Choudhary, K., Ramayapally, V.S., Vaidyanathan, S., and Hupkes, D. (August, January 31). Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2), Vienna, Austria. Available online: https:\/\/aclanthology.org\/2025.gem-1.33\/."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alsobeh, A., and Woodward, B. (2023, January 11\u201314). AI as a Partner in Learning: A Novel Student-in-the-Loop Framework for Enhanced Student Engagement and Outcomes in Higher Education. Proceedings of the 24th Annual Conference on Information Technology Education, New York, NY, USA.","DOI":"10.1145\/3585059.3611405"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Al-Ahmad, B., Alsobeh, A., Meqdadi, O., and Shaikh, N. (2025). A Student-Centric Evaluation Survey to Explore the Impact of LLMs on UML Modeling. Information, 16.","DOI":"10.20944\/preprints202505.2054.v1"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Alhazeem, E., Alsobeh, A., and Al-Ahmad, B. (2024, January 10\u201312). Enhancing Software Engineering Education through AI: An Empirical Study of Tree-Based Machine Learning for Defect Prediction. Proceedings of the 25th Annual Conference on Information Technology Education, New York, NY, USA.","DOI":"10.1145\/3686852.3686881"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kim, S., and Oh, D. (2025). Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks?. Appl. Sci., 15.","DOI":"10.3390\/app15062971"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1272229","DOI":"10.3389\/feduc.2023.1272229","article-title":"Is GPT-4 a Reliable Rater? Evaluating Consistency in GPT-4\u2019s Text Ratings","volume":"8","author":"Hackl","year":"2023","journal-title":"Front. Educ."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Seo, H., Hwang, T., Jung, J., Namgoong, H., Lee, J., and Jung, S. (2025). Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy. Appl. Sci., 15.","DOI":"10.3390\/app15020671"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wu, X.-K., Chen, M., Li, W., Wang, R., Lu, L., Liu, J., Hwang, K., Hao, Y., Pan, Y., and Meng, Q. (2025). LLM Fine-Tuning: Concepts, Opportunities, and Challenges. Big Data Cogn. Comput., 9.","DOI":"10.3390\/bdcc9040087"},{"key":"ref_19","unstructured":"Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2025, September 11). Mistral 7B. Available online: https:\/\/arxiv.org\/abs\/2310.06825."},{"key":"ref_20","unstructured":"Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Anghel, C., Anghel, A.A., Pecheanu, E., Susnea, I., Cocu, A., and Istrate, A. (2025). Multi-Model Dialectical Evaluation of LLM Reasoning Chains: A Structured Framework with Dual Scoring Agents. Informatics, 12.","DOI":"10.3390\/informatics12030076"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhao, L., Gao, W., and Fang, J. (2024). Optimizing Large Language Models on Multi Core CPUs: A Case Study of the BERT Model. Appl. Sci., 14.","DOI":"10.3390\/app14062364"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2020, January 16\u201320). Transformers: State-of-the-Art Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online. Available online: https:\/\/aclanthology.org\/2020.emnlp-demos.6.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_24","unstructured":"Face, H. (2025, September 11). PEFT: Parameter-Efficient Fine-Tuning Library. Available online: https:\/\/huggingface.co\/docs\/peft."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ain, Q.U., Chatti, M.A., Bakar, K.G.C., Joarder, S., and Alatrash, R. (2023). Automatic Construction of Educational Knowledge Graphs: A Word Embedding-Based Approach. Information, 14.","DOI":"10.3390\/info14100526"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Varam, D., Khalil, L., and Shanableh, T. (2024). On-Edge Deployment of Vision Transformers for Medical Diagnostics Using the Kvasir-Capsule. Appl. Sci., 14.","DOI":"10.3390\/app14188115"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wei, Y., Zhang, R., Zhang, J., Qi, D., and Cui, W. (2025). Research on Intelligent Grading of Physics Problems Based on Large Language Models. Educ. Sci., 15.","DOI":"10.3390\/educsci15020116"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Pingua, B., Sahoo, A., Kandpal, M., Murmu, D., Rautaray, J., Barik, R.K., and Saikia, M.J. (2025). Medical LLMs: Fine-Tuning vs. Retrieval-Augmented Generation. Bioengineering, 12.","DOI":"10.3390\/bioengineering12070687"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Almasre, M. (2024). Development and Evaluation of a Custom GPT for the Assessment of Students\u2019 Designs in a Typography Course. Educ. Sci., 14.","DOI":"10.3390\/educsci14020148"},{"key":"ref_30","first-page":"i-21","article-title":"Automated Essay Scoring with E-rater\u00ae V.2","volume":"2004","author":"Attali","year":"2004","journal-title":"ETS Res. Rep. Ser."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wang, Y., Wang, C., Li, R., and Lin, H. (2022, January 10\u201315). On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Seattle, WA, USA. Available online: https:\/\/aclanthology.org\/2022.naacl-main.249.","DOI":"10.18653\/v1\/2022.naacl-main.249"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2497","DOI":"10.1007\/s10462-021-10068-2","article-title":"An Automated Essay Scoring Systems: A Systematic Literature Review","volume":"55","author":"Ramesh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1007\/s10462-024-11017-5","article-title":"A Survey on Deep Learning-Based Automated Essay Scoring and Feedback Generation","volume":"58","author":"Misgna","year":"2025","journal-title":"Artif. Intell. Rev."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/10\/431\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T07:40:56Z","timestamp":1760514056000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/10\/431"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,11]]},"references-count":33,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["computers14100431"],"URL":"https:\/\/doi.org\/10.3390\/computers14100431","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,11]]}}}