{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T04:46:19Z","timestamp":1782362779510,"version":"3.54.5"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>With the recent rapid increase in digitization across all major industries, acquiring programming skills has increased the demand for introductory programming courses. This has further resulted in universities integrating programming courses into a wide range of curricula, including not only technical studies but also business and management fields of study. Consequently, additional resources are needed for teaching, grading, and tutoring students with diverse educational backgrounds and skills. As part of this, Automated Programming Assessment Systems (APASs) have emerged, providing scalable and high-quality assessment systems with efficient evaluation and instant feedback. Commonly, APASs heavily rely on predefined unit tests for generating feedback, which often limits the scope and level of detail of feedback that can be provided to students. With the rise of Large Language Models (LLMs) in recent years, new opportunities have emerged as these technologies can enhance feedback quality and personalization. To investigate how different feedback mechanisms in APASs are perceived by students, and how effective they are in supporting problem-solving, we have conducted a large-scale study with over 200 students from 2 different universities. Specifically, we compare baseline Compiler Feedback, standard Unit Test Feedback, and advanced LLM-based Feedback regarding perceived quality and impact on student performance. Results indicate that while students rate Unit Test Feedback as the most helpful, AI-generated feedback leads to significantly better performances. These findings suggest combining unit tests and AI-driven guidance to optimize automated feedback mechanisms and improve learning outcomes in programming education.<\/jats:p>","DOI":"10.1145\/3773911","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T16:05:18Z","timestamp":1761667518000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["A Survey on Feedback Types in Automated Programming Assessment Systems"],"prefix":"10.1145","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-5959-4936","authenticated-orcid":false,"given":"Eduard","family":"Frankford","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Innsbruck, Innsbruck, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5513-1073","authenticated-orcid":false,"given":"Tobias","family":"Antensteiner","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Innsbruck, Innsbruck, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2672-9230","authenticated-orcid":false,"given":"Michael","family":"Vierhauser","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Innsbruck, Innsbruck, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9464-5080","authenticated-orcid":false,"given":"Clemens","family":"Sauerwein","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Innsbruck, Innsbruck, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7685-6273","authenticated-orcid":false,"given":"Vivien","family":"Wallner","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence and Human Interfaces, University of Salzburg, Salzburg, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0905-6791","authenticated-orcid":false,"given":"Iris","family":"Groher","sequence":"additional","affiliation":[{"name":"Institute of Business Informatics\u2013Software Engineering, Johannes Kepler University Linz, Linz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4659-2385","authenticated-orcid":false,"given":"Reinhold","family":"Pl\u00f6sch","sequence":"additional","affiliation":[{"name":"Institute of Business Informatics\u2013Software Engineering, Johannes Kepler University\rLinz, Linz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7093-4341","authenticated-orcid":false,"given":"Ruth","family":"Breu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Innsbruck, Innsbruck, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,12,16]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00039"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381703"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993400500150747"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2010.2098442"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-69132-7_29"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653543"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66610-5_57"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2015.1024607"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2021.100027"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3024102"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSEET49119.2020.9206231"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.3102\/0034654314564881"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1964.10490181"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/1384271.1384371"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.28945\/2304"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.4995\/HEAd22.2022.14587"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639474.3640061"},{"key":"e_1_3_2_19_2","unstructured":"GitHub. 2024. GitHub CoPilot AI Editor. Retrieved January 10 2025 from https:\/\/github.com\/features\/copilot"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2445196.2445368"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1119\/1.1634963"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.2982292"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/1930464.1930480"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231711"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-52240-7_26"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159602"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381701"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483441"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2810041"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.169"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2990980"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3636515"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2014.58"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.31033\/ijemr.8.3.26"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3513140"},{"key":"e_1_3_2_36_2","unstructured":"Vreda Pieterse. 2013. Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research (CSERC \u201913) 45\u201356."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSEET62301.2024.10662988"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2021.100029"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1177\/2158244019885136"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/52.3-4.591"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462195"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/EDUCON.2016.7474573"},{"key":"e_1_3_2_43_2","unstructured":"Samuel Truniger. 2023. Codeboard: Verbesserung und Erweiterung der automatisierten Hilfestellungen. Bachelor\u2019s thesis. Z\u00fcrcher Hochschule f\u00fcr Angewandte Wissenschaften Winterthur Switzerland."},{"key":"e_1_3_2_44_2","volume-title":"Exploratory Data Analysis","author":"Tukey John W.","year":"1977","unstructured":"John W. Tukey. 1977. Exploratory Data Analysis. Addison-Wesley."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21974"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1002\/0471028959.sof142"},{"key":"e_1_3_2_47_2","volume-title":"Proceedings of the International Conference on Information Systems","author":"Vial Gregory","year":"2018","unstructured":"Gregory Vial and Bogdan Negoita. 2018. Teaching programming to non-programmers: The case of Python and Jupyter Notebooks. In Proceedings of the International Conference on Information Systems."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.5555\/2349018"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383923.3383939"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22385"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2018.08.191"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3773911","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T08:24:06Z","timestamp":1765873446000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3773911"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,16]]},"references-count":50,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3773911"],"URL":"https:\/\/doi.org\/10.1145\/3773911","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,16]]},"assertion":[{"value":"2025-04-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}