{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,22]],"date-time":"2026-11-22T17:18:55Z","timestamp":1795367935708,"version":"4.10.3"},"reference-count":193,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,2,19]],"date-time":"2024-02-19T00:00:00Z","timestamp":1708300800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>We conducted a systematic literature review on automated grading and feedback tools for programming education. We analysed 121 research papers from 2017 to 2021 inclusive and categorised them based on skills assessed, approach, language paradigm, degree of automation, and evaluation techniques. Most papers assess the correctness of assignments in object-oriented languages. Typically, these tools use a dynamic technique, primarily unit testing, to provide grades and feedback to the students or static analysis techniques to compare a submission with a reference solution or with a set of correct student submissions. However, these techniques\u2019 feedback is often limited to whether the unit tests have passed or failed, the expected and actual output, or how they differ from the reference solution. Furthermore, few tools assess the maintainability, readability, or documentation of the source code, with most using static analysis techniques, such as code quality metrics, in conjunction with grading correctness. Additionally, we found that most tools offered fully automated assessment to allow for near-instantaneous feedback and multiple resubmissions, which can increase student satisfaction and provide them with more opportunities to succeed. In terms of techniques used to evaluate the tools\u2019 performance, most papers primarily use student surveys or compare the automatic assessment tools to grades or feedback provided by human graders. However, because the evaluation dataset is frequently unavailable, it is more difficult to reproduce results and compare tools to a collection of common assignments.<\/jats:p>","DOI":"10.1145\/3636515","type":"journal-article","created":{"date-parts":[[2023,12,13]],"date-time":"2023-12-13T06:38:37Z","timestamp":1702449517000},"page":"1-43","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":269,"title":["Automated Grading and Feedback Tools for Programming Education: A Systematic Review"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5915-9153","authenticated-orcid":false,"given":"Marcus","family":"Messer","sequence":"first","affiliation":[{"name":"King\u2019s College London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6086-2479","authenticated-orcid":false,"given":"Neil C. C.","family":"Brown","sequence":"additional","affiliation":[{"name":"King\u2019s College London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0544-2003","authenticated-orcid":false,"given":"Michael","family":"K\u00f6lling","sequence":"additional","affiliation":[{"name":"King\u2019s College London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4933-0073","authenticated-orcid":false,"given":"Miaojing","family":"Shi","sequence":"additional","affiliation":[{"name":"Tongji University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,2,19]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993400500150747"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2019.0100328"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3017680.3017761"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2008.130"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2831425.2831427"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.5555\/318773.318944"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844584"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3174781.3174785"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.295895"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICKII51822.2021.9574713"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1080\/13538320802278347"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2017.8190524"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1085313.1085331"},{"key":"e_1_3_4_15_2","unstructured":"Juan Carlos Rodr\u00edguez del Pino Enrique Rubio Royo and Zen\u00f3n Hern\u00e1ndez Figueroa. 2012. A virtual programming lab for Moodle with automatic assessment and anti-plagiarism features. In Proceedings of The 2012 Internacional Conference on e-Learning e-Business Entreprise Information Systems & e-Government. Retrieved from http:\/\/hdl.handle.net\/10553\/9773"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341525.3387384"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3441636.3442309"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1080\/00405841.2016.1148989"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-59419-0_57"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/1163405.1163409"},{"key":"e_1_3_4_21_2","first-page":"338","volume-title":"Proceedings of the 13th International Conference on Educational Data Mining","author":"Efremov Aleksandr","year":"2020","unstructured":"Aleksandr Efremov, Ahana Ghosh, and Adish Singla. 2020. Zero-shot learning of hint policy via reinforcement learning and program synthesis. In Proceedings of the 13th International Conference on Educational Data Mining. 338\u2013394."},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.3390\/info11090430"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734412"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3231002"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.5555\/540137"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524610.3527909"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569817"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/367415.367422"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/1930464.1930480"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2729094.2742615"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.5121\/ijsea.2010.1302"},{"key":"e_1_3_4_32_2","volume-title":"Testing Computer Software","author":"Kaner Cem","year":"1999","unstructured":"Cem Kaner, Jack Falk, and Hung Q. Nguyen. 1999. Testing Computer Software. John Wiley & Sons."},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3059009.3059061"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231711"},{"issue":"5","key":"e_1_3_4_35_2","first-page":"223","article-title":"Importance of software documentation","volume":"10","author":"Kipyegen Noela J.","year":"2013","unstructured":"Noela J. Kipyegen and William P. K. Korir. 2013. Importance of software documentation. International Journal of Computer Science Issues 10, 5 (092013), 223\u2013228.","journal-title":"International Journal of Computer Science Issues"},{"key":"e_1_3_4_36_2","unstructured":"B. Kitchenham and S. Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report Technical Report Ver. 2.3 EBSE Technical Report. Vol. 5 EBSE; 2007."},{"key":"e_1_3_4_37_2","unstructured":"Malcolm S. Knowles. 1975. Self-directed learning: A guide for learners and teachers. Association Press."},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993400802156400"},{"key":"e_1_3_4_39_2","first-page":"8","article-title":"The problem of teaching object-oriented programming, part 1: Languages","author":"K\u00f6lling Michael","year":"1999","unstructured":"Michael K\u00f6lling. 1999. The problem of teaching object-oriented programming, part 1: Languages. Journal of Object-Oriented Programming 11, 8 (1999), 8\u201315.","journal-title":"Journal of Object-Oriented Programming"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381701"},{"issue":"2","key":"e_1_3_4_41_2","first-page":"109","article-title":"A review of techniques in automatic programming assessment for practical skill test","volume":"10","author":"Lajis Adidah","year":"2018","unstructured":"Adidah Lajis, Shahidatul Arfah Baharudin, Diyana Ab Kadir, Nadilah Mohd Ralim, Haidawati Mohd Nasir, and Normaziah Abdul Aziz. 2018. A review of techniques in automatic programming assessment for practical skill test. Journal of Telecommunication, Electronic and Computer Engineering 10, 2\u20135 (072018), 109\u2013113. Retrieved from https:\/\/jtec.utem.edu.my\/jtec\/article\/view\/4394","journal-title":"Journal of Telecommunication, Electronic and Computer Engineering"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3478431.3499372"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.288"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2716950"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1976.233837"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","unstructured":"Marcus Messer. 2022. Automated Grading and Feedback Tools: A Systematic Review. Open Science Foundation Preregistrations. DOI:10.17605\/OSF.IO\/VXTF9","DOI":"10.17605\/OSF.IO\/VXTF9"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587102.3588822"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203880869-13"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCI54379.2022.9740769"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587102.3588794"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1186\/S13643-016-0384-4"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3554697"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-021-01626-4"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3513140"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3059009.3059026"},{"key":"e_1_3_4_56_2","first-page":"1093","volume-title":"Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research)","volume":"37","author":"Piech Chris","year":"2015","unstructured":"Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. 2015. Learning program embeddings to propagate feedback on student code. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research). Francis Bach and David Blei (Eds.), Vol. 37. PMLR, Lille, 1093\u20131102. Retrieved from https:\/\/proceedings.mlr.press\/v37\/piech15.html"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077618"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.3390\/app10082973"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3519312"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/1053331.1053354"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587102.3588792"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICoDSE53690.2021.9648464"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459995"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462195"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSEET.2016.48"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1046\/j.1365-2575.2002.00117.x"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/1151588.1151605"},{"key":"e_1_3_4_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/2999541.2999555"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2013.6613836"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21974"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287463"},{"key":"e_1_3_4_72_2","first-page":"495","article-title":"Automatic assessment of the design quality of Python programs with personalized feedback","author":"Walker Orr","year":"2021","unstructured":"Orr Walker and Nathaniel Russell. 2021. Automatic assessment of the design quality of Python programs with personalized feedback. In Proceedings of the 14th International Conference on Educational Data Mining. 495\u2013501.","journal-title":"Proceedings of the 14th International Conference on Educational Data Mining"},{"key":"e_1_3_4_73_2","article-title":"Combining dynamic and static analysis for automated grading SQL statements","volume":"5","author":"Wang Jinshui","year":"2020","unstructured":"Jinshui Wang, Yunpeng Zhao, Zhengyi Tang, and Zhenchang Xing. 2020. Combining dynamic and static analysis for automated grading SQL statements. Taiwan Ubiquitous Information 5, 4 (2020), 179\u2013190.","journal-title":"Taiwan Ubiquitous Information"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33642-3_25"},{"key":"e_1_3_5_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00039"},{"key":"e_1_3_5_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381703"},{"key":"e_1_3_5_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236777"},{"key":"e_1_3_5_5_2","doi-asserted-by":"publisher","DOI":"10.2312\/eged.20191025"},{"key":"e_1_3_5_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2020.2981786"},{"key":"e_1_3_5_7_2","doi-asserted-by":"publisher","DOI":"10.11591\/ijeecs.v20.i1.pp454-464"},{"key":"e_1_3_5_8_2","doi-asserted-by":"publisher","DOI":"10.18517\/ijaseit.8.4.6416"},{"key":"e_1_3_5_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICoDSE53690.2021.9648439"},{"key":"e_1_3_5_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3426425.3426935"},{"key":"e_1_3_5_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSCI51800.2020.00177"},{"key":"e_1_3_5_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66610-5_57"},{"key":"e_1_3_5_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-017-0612-y"},{"key":"e_1_3_5_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/WEEF\/GEDC53299.2021.9657355"},{"key":"e_1_3_5_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/BotSE52550.2021.00010"},{"key":"e_1_3_5_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-35343-8_22"},{"key":"e_1_3_5_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3110248"},{"key":"e_1_3_5_18_2","doi-asserted-by":"publisher","DOI":"10.23919\/CISTI.2018.8399261"},{"key":"e_1_3_5_19_2","doi-asserted-by":"publisher","DOI":"10.1111\/exsy.12531"},{"key":"e_1_3_5_20_2","first-page":"443","volume-title":"Proceedings of the 31st International Florida Artificial Intelligence Research Society Conference","author":"Carmichael Ted","year":"2018","unstructured":"Ted Carmichael, Mary Jean Blink, John C. Stamper, and Elizabeth Gieske. 2018. Linkage objects for generalized instruction in coding (LOGIC). In Proceedings of the 31st International Florida Artificial Intelligence Research Society Conference. Keith Brawner and Vasile Rus (Eds.), AAAI Press, 443\u2013446. Retrieved from https:\/\/aaai.org\/ocs\/index.php\/FLAIRS\/FLAIRS18\/paper\/view\/17702"},{"key":"e_1_3_5_21_2","doi-asserted-by":"publisher","DOI":"10.6688\/JISE.201809_34(5).0006"},{"key":"e_1_3_5_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICKII51822.2021.9574713"},{"key":"e_1_3_5_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3099023.3099065"},{"key":"e_1_3_5_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEET52601.2021.00015"},{"key":"e_1_3_5_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSEET49119.2020.9206231"},{"key":"e_1_3_5_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2018.2876249"},{"key":"e_1_3_5_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287511"},{"key":"e_1_3_5_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432417"},{"key":"e_1_3_5_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372356.3372357"},{"key":"e_1_3_5_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-58515-4_2"},{"key":"e_1_3_5_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/EAEEIE50507.2021.9530951"},{"key":"e_1_3_5_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/CogMI48466.2019.00018"},{"key":"e_1_3_5_33_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22317"},{"key":"e_1_3_5_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3441636.3442309"},{"key":"e_1_3_5_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/LWMOOCS50143.2020.9234387"},{"key":"e_1_3_5_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/EduHPC54835.2021.00010"},{"key":"e_1_3_5_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290511.3290528"},{"key":"e_1_3_5_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-59419-0_57"},{"key":"e_1_3_5_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381707"},{"key":"e_1_3_5_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-50436-6_45#editor-information"},{"key":"e_1_3_5_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3358711.3361626"},{"key":"e_1_3_5_42_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22043"},{"key":"e_1_3_5_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-015-0080-x"},{"key":"e_1_3_5_44_2","article-title":"Developing applications to automatically grade introductory visual basic courses","author":"Gerdes John","year":"2017","unstructured":"John Gerdes. 2017. Developing applications to automatically grade introductory visual basic courses. In Proceedings of the Americas Conference on Information Systems.","journal-title":"Proceedings of the Americas Conference on Information Systems"},{"key":"e_1_3_5_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3296979.3192387"},{"key":"e_1_3_5_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58459-7_12"},{"key":"e_1_3_5_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159502"},{"key":"e_1_3_5_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341719"},{"key":"e_1_3_5_49_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2020.1860408"},{"key":"e_1_3_5_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE43999.2019.9028686"},{"key":"e_1_3_5_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2019.8818922"},{"key":"e_1_3_5_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437914.3437973"},{"key":"e_1_3_5_53_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22356"},{"key":"e_1_3_5_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cl.2018.01.004"},{"key":"e_1_3_5_55_2","volume-title":"Post-proceedings of the 10th Seminar on Advanced Techniques and Tools for Software Evolution","author":"Jansen Julian","year":"2017","unstructured":"Julian Jansen, Ana Oprescu, and Magiel Bruntink. 2017. The impact of automated code quality feedback in programming education. In Post-proceedings of the 10th Seminar on Advanced Techniques and Tools for Software Evolution. Vol. 210."},{"key":"e_1_3_5_56_2","doi-asserted-by":"publisher","DOI":"10.1564\/tme_v24.3.03"},{"key":"e_1_3_5_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197091.3197094"},{"key":"e_1_3_5_58_2","unstructured":"Angelika Kaplan Jan Keim Yves R. Schneider Maximilian Walter Dominik Werle Anne Koziolek and Ralf Reussner. 2020. Teaching Programming at Scale. Software Engineering im Unterricht der Hochschulen (SEUH\u201920). Retrieved from https:\/\/ceur-ws.org\/Vol-2531\/paper01.pdf"},{"key":"e_1_3_5_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE49875.2021.9637193"},{"key":"e_1_3_5_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3304221.3319792"},{"key":"e_1_3_5_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.110905"},{"key":"e_1_3_5_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428029.3428043"},{"key":"e_1_3_5_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432526"},{"key":"e_1_3_5_64_2","doi-asserted-by":"publisher","DOI":"10.25304\/RLT.V25.1945"},{"key":"e_1_3_5_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159602"},{"key":"e_1_3_5_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-61425-0_14"},{"key":"e_1_3_5_67_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22464"},{"key":"e_1_3_5_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432463"},{"key":"e_1_3_5_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3276528"},{"key":"e_1_3_5_70_2","doi-asserted-by":"publisher","DOI":"10.1111\/jcal.12238"},{"key":"e_1_3_5_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366921"},{"key":"e_1_3_5_72_2","doi-asserted-by":"publisher","DOI":"10.28945\/4150"},{"key":"e_1_3_5_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373165.3373176"},{"key":"e_1_3_5_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287503"},{"key":"e_1_3_5_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICICE49024.2019.9117506"},{"key":"e_1_3_5_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEET.2019.00022"},{"key":"e_1_3_5_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678837"},{"key":"e_1_3_5_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366954"},{"key":"e_1_3_5_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287505"},{"key":"e_1_3_5_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366947"},{"key":"e_1_3_5_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.169"},{"key":"e_1_3_5_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2990980"},{"key":"e_1_3_5_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66610-5_12"},{"key":"e_1_3_5_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIUCC52538.2021.9447629"},{"key":"e_1_3_5_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICS51289.2020.00099"},{"key":"e_1_3_5_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2018.8658833"},{"key":"e_1_3_5_87_2","unstructured":"Norbert Oster Marius Kamp and Michael Philippsen. 2017. AuDoscore: Automatic Grading of Java or Scala Homework. Automatische Bewertung von Programmieraufgaben (ABP\u201917)."},{"key":"e_1_3_5_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/3059009.3059026"},{"key":"e_1_3_5_89_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22289"},{"key":"e_1_3_5_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEET52601.2021.00032"},{"key":"e_1_3_5_91_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2021.100029"},{"key":"e_1_3_5_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITHET46829.2019.8937377"},{"key":"e_1_3_5_93_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-94505-7_19"},{"key":"e_1_3_5_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE43999.2019.9028450"},{"key":"e_1_3_5_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE43999.2019.9028418"},{"issue":"3","key":"e_1_3_5_96_2","first-page":"174","article-title":"Using an interactive software tool for the formative and summative evaluation in a computer programming course: An experience report","volume":"22","author":"Restrepo-Calle Felipe","year":"2020","unstructured":"Felipe Restrepo-Calle, Jhon J. Ram\u00edrez-Echeverry, and Fabio A. Gonz\u00e1lez. 2020. Using an interactive software tool for the formative and summative evaluation in a computer programming course: An experience report. Global Journal of Engineering Education 22, 3 (2020), 174\u2013185.","journal-title":"Global Journal of Engineering Education"},{"key":"e_1_3_5_97_2","doi-asserted-by":"publisher","unstructured":"Arthur Rump Ansgar Fehnker and Angelika Mader. 2021. Automated assessment of learning objectives in programming assignments. Intelligent Tutoring Systems (ITS\u201921). Lecture Notes in Computer Science 299\u2013309. DOI:10.1007\/978-3-030-80421-3_33","DOI":"10.1007\/978-3-030-80421-3_33"},{"key":"e_1_3_5_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231644.3231708"},{"key":"e_1_3_5_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICoDSE53690.2021.9648464"},{"key":"e_1_3_5_100_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93846-2_60"},{"key":"e_1_3_5_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341525.3387417"},{"key":"e_1_3_5_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939696"},{"key":"e_1_3_5_103_2","volume-title":"Proceedings of the Impact Papers at EC-TEL 2020","author":"Soll Marcus","year":"2020","unstructured":"Marcus Soll, Melf Johannsen, and Chris Biemann. 2020. Enhancing a theory-focused course through the introduction of automatically assessed programming exercises-lessons learned. In Proceedings of the Impact Papers at EC-TEL 2020. Retrieved from https:\/\/ceur-ws.org\/Vol-2676\/paper6.pdf"},{"key":"e_1_3_5_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468598"},{"key":"e_1_3_5_105_2","doi-asserted-by":"publisher","DOI":"10.5220\/0007711000620069"},{"key":"e_1_3_5_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053187"},{"key":"e_1_3_5_107_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2017.8103457"},{"key":"e_1_3_5_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338147.3338187"},{"key":"e_1_3_5_109_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287463"},{"key":"e_1_3_5_110_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.22021"},{"key":"e_1_3_5_111_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472673.3473961"},{"key":"e_1_3_5_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/3486001.3486228"},{"key":"e_1_3_5_113_2","article-title":"Combining dynamic and static analysis for automated grading SQL statements","volume":"5","author":"Wang Jinshui","year":"2020","unstructured":"Jinshui Wang, Yunpeng Zhao, Zhengyi Tang, and Zhenchang Xing. 2020. Combining dynamic and static analysis for automated grading SQL statements. Journal of Network Intelligence 5, 4 (2020), 179\u2013190.","journal-title":"Journal of Network Intelligence"},{"key":"e_1_3_5_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/3192366.3192384"},{"key":"e_1_3_5_115_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30952-7_63"},{"key":"e_1_3_5_116_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287502"},{"key":"e_1_3_5_117_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC51239.2020.9357320"},{"key":"e_1_3_5_118_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197091.3197112"},{"key":"e_1_3_5_119_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383923.3383939"},{"key":"e_1_3_5_120_2","doi-asserted-by":"publisher","DOI":"10.1109\/TALE.2017.8252370"},{"key":"e_1_3_5_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3426431.3428654"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3636515","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3636515","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:54:11Z","timestamp":1750272851000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3636515"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,19]]},"references-count":193,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3636515"],"URL":"https:\/\/doi.org\/10.1145\/3636515","relation":{"has-preprint":[{"id-type":"doi","id":"10.35542\/osf.io\/wpsgk","asserted-by":"object"}]},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,19]]},"assertion":[{"value":"2023-06-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}