{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T02:18:14Z","timestamp":1774405094861,"version":"3.50.1"},"reference-count":26,"publisher":"Oxford University Press (OUP)","issue":"Supplement_1","license":[{"start":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T00:00:00Z","timestamp":1752537600000},"content-version":"vor","delay-in-days":14,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Research Program","award":["P2-0209"],"award-info":[{"award-number":["P2-0209"]}]},{"name":"Research Program","award":["V2-2272"],"award-info":[{"award-number":["V2-2272"]}]},{"name":"Young Researcher","award":["57111"],"award-info":[{"award-number":["57111"]}]},{"DOI":"10.13039\/501100004329","name":"Slovenian Research and Innovation Agency","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Providing students with individualized feedback through assignments is a cornerstone of education that supports their learning and development. Studies have shown that timely, high-quality feedback plays a critical role in improving learning outcomes. However, providing personalized feedback on a large scale in classes with large numbers of students is often impractical due to the significant time and effort required. Recent advances in natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback. These technologies can reduce the workload of course staff while improving student satisfaction and learning outcomes. Their successful implementation, however, requires thorough evaluation and validation in real classrooms.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We present the results of a practical evaluation of LLM-based graders for written assignments in the 2024\/25 iteration of the Introduction to Bioinformatics course at the University of Ljubljana. Over the course of the semester, more than 100 students answered 36 text-based questions, most of which were automatically graded using LLMs. In a blind study, students received feedback from both LLMs and human teaching assistants (TAs) without knowing the source, and later rated the quality of the feedback. We conducted a systematic evaluation of six commercial and open-source LLMs and compared their grading performance with human TAs. Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders. Our results also suggest that open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btaf196","type":"journal-article","created":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T13:02:11Z","timestamp":1752584531000},"page":"i21-i29","source":"Crossref","is-referenced-by-count":8,"title":["Automated assignment grading with large language models: insights from a bioinformatics course"],"prefix":"10.1093","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6462-9372","authenticated-orcid":false,"given":"Pavlin G","family":"Poli\u010dar","sequence":"first","affiliation":[{"name":"Faculty of Computer and Information Science, University of Ljubljana , Ve\u010dna pot 113 , 1000 Ljubljana,","place":["Slovenia"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-0796-8985","authenticated-orcid":false,"given":"Martin","family":"\u0160pendl","sequence":"additional","affiliation":[{"name":"Faculty of Computer and Information Science, University of Ljubljana , Ve\u010dna pot 113 , 1000 Ljubljana,","place":["Slovenia"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4888-7256","authenticated-orcid":false,"given":"Toma\u017e","family":"Curk","sequence":"additional","affiliation":[{"name":"Faculty of Computer and Information Science, University of Ljubljana , Ve\u010dna pot 113 , 1000 Ljubljana,","place":["Slovenia"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5864-7056","authenticated-orcid":false,"given":"Bla\u017e","family":"Zupan","sequence":"additional","affiliation":[{"name":"Faculty of Computer and Information Science, University of Ljubljana , Ve\u010dna pot 113 , 1000 Ljubljana,","place":["Slovenia"]},{"name":"Department of Education, Innovation and Technology, Baylor College of Medicine , 1 Baylor Plz , Houston, TX 77030,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2025,7,15]]},"reference":[{"key":"2025071509020945900_btaf196-B1","author":"AI@Meta"},{"key":"2025071509020945900_btaf196-B2","author":"Chiang","year":"2023"},{"key":"2025071509020945900_btaf196-B3","first-page":"2489","author":"Chiang","year":"2024"},{"key":"2025071509020945900_btaf196-B4","first-page":"323","author":"Dai","year":"2023"},{"key":"2025071509020945900_btaf196-B5","author":"OpenAI.","year":"2024"},{"key":"2025071509020945900_btaf196-B6","author":"Impey"},{"key":"2025071509020945900_btaf196-B7","first-page":"3097","author":"Jauhiainen","year":"2024"},{"key":"2025071509020945900_btaf196-B8","author":"Jin"},{"key":"2025071509020945900_btaf196-B9","author":"Kaddour","year":"2023"},{"key":"2025071509020945900_btaf196-B10","author":"Katuka","year":"2024"},{"key":"2025071509020945900_btaf196-B11","first-page":"1023","article-title":"The failure of a decision support system: inconsistency in test grading by teachers","volume":"18","author":"Klein","year":"2002","journal-title":"Teach Educ"},{"key":"2025071509020945900_btaf196-B12","first-page":"143","author":"Kostic","year":"2024"},{"key":"2025071509020945900_btaf196-B13","volume-title":"Doing Bayesian Data Analysis","author":"Kruschke","year":"2015","edition":"2nd edn"},{"key":"2025071509020945900_btaf196-B14","author":"Liu"},{"key":"2025071509020945900_btaf196-B15","author":"Lyu","year":"2024"},{"key":"2025071509020945900_btaf196-B16","doi-asserted-by":"crossref","first-page":"102173","DOI":"10.1016\/j.ijer.2023.102173","article-title":"Teacher burnout and physical health: a systematic review","volume":"119","author":"Madigan","year":"2023","journal-title":"Int J Educ Res"},{"key":"2025071509020945900_btaf196-B17","author":"Minaee","year":"2024"},{"key":"2025071509020945900_btaf196-B18","first-page":"752","author":"Mohler","year":"2011"},{"key":"2025071509020945900_btaf196-B19","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1007\/BF01419938","article-title":"The use of the computer in analyzing student essays","volume":"14","author":"Page","year":"1968","journal-title":"Int Rev Educ"},{"key":"2025071509020945900_btaf196-B20","doi-asserted-by":"crossref","first-page":"104236","DOI":"10.1016\/j.nedt.2019.104236","article-title":"What are students\u2019 needs and preferences for academic feedback in higher education? A systematic review","volume":"85","author":"Paterson","year":"2020","journal-title":"Nurse Educ Today"},{"key":"2025071509020945900_btaf196-B21","doi-asserted-by":"crossref","first-page":"i20","DOI":"10.1093\/bioinformatics\/btae208","article-title":"Teaching bioinformatics through the analysis of SARS-CoV-2: project-based training for computer science students","volume":"40","author":"Poli\u010dar","year":"2024","journal-title":"Bioinformatics"},{"key":"2025071509020945900_btaf196-B22","first-page":"159","author":"Riordan","year":"2017"},{"key":"2025071509020945900_btaf196-B23","author":"Stan Development Team","year":"2025"},{"key":"2025071509020945900_btaf196-B24","author":"Team","year":"2024"},{"key":"2025071509020945900_btaf196-B25","author":"Wang","year":"2025"},{"key":"2025071509020945900_btaf196-B26","author":"Yadav","year":"2023"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/41\/Supplement_1\/i21\/63745528\/btaf196.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/41\/Supplement_1\/i21\/63745528\/btaf196.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T13:02:13Z","timestamp":1752584533000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/41\/Supplement_1\/i21\/8199383"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,1]]},"references-count":26,"journal-issue":{"issue":"Supplement_1","published-print":{"date-parts":[[2025,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btaf196","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,7]]},"published":{"date-parts":[[2025,7,1]]}}}