{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T05:57:17Z","timestamp":1770357437243,"version":"3.49.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,5,10]],"date-time":"2021-05-10T00:00:00Z","timestamp":1620604800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2021,9,30]]},"abstract":"<jats:p>Autograding systems are being increasingly deployed to meet the challenges of teaching programming at scale. Studies show that formative feedback can greatly help novices learn programming. This work extends an autograder, enabling it to provide formative feedback on programming assignment submissions. Our methodology starts with the design of a knowledge map, which is the set of concepts and skills that are necessary to complete an assignment, followed by the design of the assignment and that of a comprehensive test suite for identifying logical errors in the submitted code. Test cases are used to test the student submissions and learn classes of common errors. For each assignment, we train a classifier that automatically categorizes errors in a submission based on the outcome of the test suite. The instructor maps the errors to corresponding concepts and skills and writes hints to help students find their misconceptions and mistakes. We apply this methodology to two assignments in our Introduction to Computer Science course and find that the automatic error categorization has a 90% average accuracy. We report and compare data from two semesters, one semester when hints are given for the two assignments and one when hints are not given. Results show that the percentage of students who successfully complete the assignments after an initial erroneous submission is three times greater when hints are given compared to when hints are not given. However, on average, even when hints are provided, almost half of the students fail to correct their code so that it passes all the test cases. The initial implementation of the framework focuses on the functional correctness of the programs as reflected by the outcome of the test cases. In our future work, we will explore other kinds of feedback and approaches to automatically generate feedback to better serve the educational needs of the students.<\/jats:p>","DOI":"10.1145\/3445983","type":"journal-article","created":{"date-parts":[[2021,5,10]],"date-time":"2021-05-10T15:32:57Z","timestamp":1620660777000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["CSF: Formative Feedback in Autograding"],"prefix":"10.1145","volume":"21","author":[{"given":"Georgiana","family":"Haldeman","sequence":"first","affiliation":[{"name":"Rutgers University, Computer Science Department, Piscataway, NJ, USA"}]},{"given":"Monica","family":"Babe\u015f-Vroman","sequence":"additional","affiliation":[{"name":"Rutgers University, Computer Science Department, Piscataway, NJ, USA"}]},{"given":"Andrew","family":"Tjang","sequence":"additional","affiliation":[{"name":"Rutgers University, Computer Science Department, Piscataway, NJ, USA"}]},{"given":"Thu D.","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Rutgers University, Computer Science Department, Piscataway, NJ, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,5,10]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"National Center for Women & Information Technology. n.d. Projected computing jobs and CIS degrees earned. Retrieved from http:\/\/www.ncwit.org\/sites\/default\/files\/file_type\/usnatgraphic2022projections_10132014.pdf.  National Center for Women & Information Technology. n.d. Projected computing jobs and CIS degrees earned. Retrieved from http:\/\/www.ncwit.org\/sites\/default\/files\/file_type\/usnatgraphic2022projections_10132014.pdf."},{"key":"e_1_2_1_2_1","volume-title":"n.d. Generation CS: Computer science undergraduate enrollments surge since","author":"Computing Research Association","year":"2006"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1597849.1384371"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2011.2"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2899415.2899422"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568023"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045235"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820011"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3051467"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.3102\/0034654307313795"},{"key":"e_1_2_1_11_1","unstructured":"Susan A. Ambrose Michael W. Bridges Michele DiPietro Marsha C. Lovett and Marie K. Norman. 2010. How Learning Works: Seven Research-Based Principles for Smart Teaching. Jossey-Bass.  Susan A. Ambrose Michael W. Bridges Michele DiPietro Marsha C. Lovett and Marie K. Norman. 2010. How Learning Works: Seven Research-Based Principles for Smart Teaching. Jossey-Bass."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339420"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/2155693"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462195"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3456126.3456142"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 2012 Annual Meeting of the American Educational Research Association. 1\u201325","author":"Brennan Karen","year":"2012"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2013.16"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 2nd Workshop on AI-supported Education for Computer Science.","author":"Le Nguyen-Thinh","year":"2014"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939696"},{"key":"e_1_2_1_20_1","volume-title":"A survey of automated programming hint generation - The HINTS framework. ArXiv abs\/1908.11566","author":"McBroom Jessica","year":"2019"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-41540-6_21"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3296979.3192384"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053187"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71--78","author":"Stamper J."},{"key":"e_1_2_1_25_1","volume-title":"Tiffany Barnes, Sebastian Gross, and Niels Pinkwart.","author":"Paa\u00dfen Benjamin","year":"2017"},{"key":"e_1_2_1_26_1","volume-title":"the 12th International Conference on Educational Data Mining. 218--227","author":"Zhi Rui","year":"2019"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-015-0080-x"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807469"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2699751"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807495"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2724660.2728672"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1930464.1930480"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 2013 Workshop on Massive Open Online Courses at the 16th Annual Conference on Artificial Intelligence in Education.","author":"Huang Jonathan","year":"2013"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950363"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753478"},{"key":"e_1_2_1_36_1","volume-title":"The 2009 International Conference on Computers in Education.","author":"Holland Jay","year":"2009"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-015-0070-z"},{"key":"e_1_2_1_38_1","volume-title":"A survey of machine learning for big code and naturalness. arXiv preprint arXiv:1709.06182","author":"Allamanis Miltiadis","year":"2017"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.2307\/249008"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3445983","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3445983","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:25Z","timestamp":1750197805000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3445983"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,10]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,9,30]]}},"alternative-id":["10.1145\/3445983"],"URL":"https:\/\/doi.org\/10.1145\/3445983","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,10]]},"assertion":[{"value":"2019-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}