{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T23:05:16Z","timestamp":1771023916510,"version":"3.50.1"},"reference-count":50,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T00:00:00Z","timestamp":1684108800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001871","name":"FCT\u2014Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia (Portuguese Foundation for Science and Technology), Portugal","doi-asserted-by":"publisher","award":["2020.04430.BD"],"award-info":[{"award-number":["2020.04430.BD"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Electronics"],"abstract":"<jats:p>Learning to program requires diligent practice and creates room for discovery, trial and error, debugging, and concept mapping. Learners must walk this long road themselves, supported by appropriate and timely feedback. Providing such feedback in programming exercises is not a humanly feasible task. Therefore, the early and steadily growing interest of computer science educators in the automated assessment of programming exercises is not surprising. The automated assessment of programming assignments has been an active area of research for over a century, and interest in it continues to grow as it adapts to new developments in computer science and the resulting changes in educational requirements. It is therefore of paramount importance to understand the work that has been performed, who has performed it, its evolution over time, the relationships between publications, its hot topics, and open problems, among others. This paper presents a bibliometric study of the field, with a particular focus on the issue of automatic feedback generation, using literature data from the Web of Science Core Collection. It includes a descriptive analysis using various bibliometric measures and data visualizations on authors, affiliations, citations, and topics. In addition, we performed a complementary analysis focusing only on the subset of publications on the specific topic of automatic feedback generation. The results are highlighted and discussed.<\/jats:p>","DOI":"10.3390\/electronics12102254","type":"journal-article","created":{"date-parts":[[2023,5,16]],"date-time":"2023-05-16T02:20:13Z","timestamp":1684203613000},"page":"2254","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Bibliometric Analysis of Automated Assessment in Programming Education: A Deeper Insight into Feedback"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0394-0527","authenticated-orcid":false,"given":"Jos\u00e9 Carlos","family":"Paiva","sequence":"first","affiliation":[{"name":"CRACS\u2014INESC TEC, 4169-007 Porto, Portugal"},{"name":"Department of Computer Science, Faculty of Sciences, University of Porto, 4169-007 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0507-7504","authenticated-orcid":false,"given":"\u00c1lvaro","family":"Figueira","sequence":"additional","affiliation":[{"name":"CRACS\u2014INESC TEC, 4169-007 Porto, Portugal"},{"name":"Department of Computer Science, Faculty of Sciences, University of Porto, 4169-007 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8409-0300","authenticated-orcid":false,"given":"Jos\u00e9 Paulo","family":"Leal","sequence":"additional","affiliation":[{"name":"CRACS\u2014INESC TEC, 4169-007 Porto, Portugal"},{"name":"Department of Computer Science, Faculty of Sciences, University of Porto, 4169-007 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1076\/csed.13.2.137.14200","article-title":"Learning and Teaching Programming: A Review and Discussion","volume":"13","author":"Robins","year":"2003","journal-title":"Comput. Sci. Educ."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"104530","DOI":"10.1016\/j.compedu.2022.104530","article-title":"Investigating the impact of research-based professional development on teacher learning and classroom practice: Findings from computer science education","volume":"186","author":"Mouza","year":"2022","journal-title":"Comput. Educ."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1145\/507758.377666","article-title":"Fully Automatic Assessment of Programming Exercises","volume":"33","author":"Saikkonen","year":"2001","journal-title":"SIGCSE Bull."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1080\/08993400500150747","article-title":"A Survey of Automated Assessment Approaches for Programming Assignments","volume":"15","year":"2005","journal-title":"Comput. Sci. Educ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1145\/3513140","article-title":"Automated Assessment in Computer Science Education: A State-of-the-Art Review","volume":"22","author":"Paiva","year":"2022","journal-title":"ACM Trans. Comput. Educ."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ihantola, P., Ahoniemi, T., Karavirta, V., and Sepp\u00e4l\u00e4, O. (2010, January 28\u201331). Review of recent systems for automatic assessment of programming assignments. Proceedings of the 10th Koli Calling International Conference on Computing Education Research\u2014Koli Calling\u201910, Koli, Finland.","DOI":"10.1145\/1930464.1930480"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Souza, D.M., Felizardo, K.R., and Barbosa, E.F. (2016, January 5\u20136). A Systematic Literature Review of Assessment Tools for Programming Assignments. Proceedings of the 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET), Dallas, TX, USA.","DOI":"10.1109\/CSEET.2016.48"},{"key":"ref_8","unstructured":"Paiva, J.C., Figueira, \u00c1., and Leal, J.P. (2022, January 21\u201323). Automated Assessment in Computer Science: A Bibliometric Analysis of the Literature. Proceedings of the Advances in Web-Based Learning\u2014ICWL 2022, Tenerife, Spain."},{"key":"ref_9","unstructured":"Andr\u00e9s, A. (2009). Measuring Academic Research, Chandos Publishing (Oxford)."},{"key":"ref_10","unstructured":"Clarivate (2023, March 19). Web of Science Core Collection. Available online: https:\/\/www.webofscience.com\/wos\/woscc\/summary\/f82ac75a-44c0-4873-a40d-c59e1e79ef4e-79ee7f28\/relevance\/1."},{"key":"ref_11","first-page":"959","article-title":"bibliometrix: An R-tool for comprehensive science mapping analysis","volume":"11","author":"Aria","year":"2017","journal-title":"J. Inf."},{"key":"ref_12","unstructured":"Wen, M., Chen, J., Wu, R., Hao, D., and Cheung, S.C. (June, January 27). Context-Aware Patch Generation for Better Automated Program Repair. Proceedings of the 40th International Conference on Software Engineering, ICSE\u201918, Gothenburg, Sweden."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1016\/j.compedu.2009.09.005","article-title":"Marking student programs using graph similarity","volume":"54","author":"Greyling","year":"2010","journal-title":"Comput. Educ."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.compedu.2011.08.015","article-title":"A distributed system for learning programming on-line","volume":"58","author":"Regueras","year":"2012","journal-title":"Comput. Educ."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Liu, K., Koyuncu, A., Kim, D., and Bissyand\u00e9, T.F. (2019, January 15\u201319). TBar: Revisiting Template-Based Automated Program Repair. Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China.","DOI":"10.1145\/3293882.3330577"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Srikant, S., and Aggarwal, V. (2014, January 24\u201327). A System to Grade Computer Programming Skills Using Machine Learning. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD\u201914, New York, NY, USA.","DOI":"10.1145\/2623330.2623377"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lemieux, C., Padhye, R., Sen, K., and Song, D. (2018, January 16\u201321). PerfFuzz: Automatically Generating Pathological Inputs. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands.","DOI":"10.1145\/3213846.3213874"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Singh, R., Gulwani, S., and Solar-Lezama, A. (2013, January 16\u201319). Automated Feedback Generation for Introductory Programming Assignments. Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI\u201913, Seattle, WA, USA.","DOI":"10.1145\/2491956.2462195"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Xiong, Y., Wang, J., Yan, R., Zhang, J., Han, S., Huang, G., and Zhang, L. (2017, January 20\u201328). Precise Condition Synthesis for Program Repair. Proceedings of the 39th International Conference on Software Engineering, ICSE\u201917, Buenos Aires, Argentina.","DOI":"10.1109\/ICSE.2017.45"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1177\/0165551504042802","article-title":"Historiographic Mapping of Knowledge Domains Literature","volume":"30","author":"Garfield","year":"2004","journal-title":"J. Inf. Sci."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1145\/3360585","article-title":"Getafix: Learning to Fix Bugs Automatically","volume":"3","author":"Bader","year":"2019","journal-title":"Proc. ACM Program. Lang."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Pettit, R.S., Homer, J., and Gee, R. (2017, January 8\u201311). Do Enhanced Compiler Error Messages Help Students? Results Inconclusive. Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, SIGCSE\u201917, Seattle, WA, USA.","DOI":"10.1145\/3017680.3017768"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., and Cohen, M. (2018, January 13\u201315). Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. Proceedings of the 2018 ACM Conference on International Computing Education Research, ICER\u201918, Espoo, Finland.","DOI":"10.1145\/3230977.3230981"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"DeNero, J., Sridhara, S., P\u00e9rez-Qui\u00f1ones, M., Nayak, A., and Leong, B. (2017, January 8\u201311). Beyond Autograding: Advances in Student Feedback Platforms. Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, SIGCSE\u201917, Seattle WA, USA.","DOI":"10.1145\/3017680.3017686"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sridhara, S., Hou, B., Lu, J., and DeNero, J. (2016, January 25\u201326). Fuzz Testing Projects in Massive Courses. Proceedings of the Third (2016) ACM Conference on Learning @ Scale, L@S\u201916, Edinburgh, UK.","DOI":"10.1145\/2876034.2876050"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Monperrus, M. (June, January 31). A Critical Review of \u201cAutomatic Patch Generation Learned from Human-Written Patches\u201d: Essay on the Problem Statement and the Evaluation of Automatic Software Repair. Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, Hyderabad, India.","DOI":"10.1145\/2568225.2568324"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2901","DOI":"10.1007\/s10664-017-9550-0","article-title":"Do Automated Program Repair Techniques Repair Hard and Important Bugs?","volume":"23","author":"Motwani","year":"2018","journal-title":"Empirical Softw. Engg."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"117","DOI":"10.15388\/infedu.2018.08","article-title":"CodeMaster\u2014Automatic Assessment and Grading of App Inventor and Snap! Programs","volume":"17","author":"Hauck","year":"2018","journal-title":"Inform. Educ."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"17","DOI":"10.15388\/infedu.2019.02","article-title":"Approaches to Assess Computational Thinking Competences Based on Code Analysis in K-12 Education: A Systematic Mapping Study","volume":"18","author":"Wangenheim","year":"2019","journal-title":"Inform. Educ."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Falkner, N., Vivian, R., Piper, D., and Falkner, K. (2014, January 5\u20138). Increasing the Effectiveness of Automated Assessment by Increasing Marking Granularity and Feedback Units. Proceedings of the 45th ACM Technical Symposium on Computer Science Education, SIGCSE\u201914, Atlanta, GA, USA.","DOI":"10.1145\/2538862.2538896"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Insa, D., and Silva, J. (2015, January 4\u20138). Semi-Automatic Assessment of Unrestrained Java Code: A Library, a DSL, and a Workbench to Assess Exams and Exercises. Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE\u201915, Vilnius, Lithuania.","DOI":"10.1145\/2729094.2742615"},{"key":"ref_32","first-page":"146","article-title":"An approach for detecting, quantifying, and visualizing the evolution of a research field: A practical application to the Fuzzy Sets Theory field","volume":"5","author":"Cobo","year":"2011","journal-title":"J. Inf."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"104094","DOI":"10.1016\/j.compedu.2020.104094","article-title":"A review of automated feedback systems for learners: Classification framework, challenges and opportunities","volume":"162","author":"Deeva","year":"2021","journal-title":"Comput. Educ."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Benotti, L., Aloi, F., Bulgarelli, F., and Gomez, M.J. (2018, January 21\u201324). The Effect of a Web-Based Coding Tool with Automatic Feedback on Students\u2019 Performance and Perceptions. Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE\u201918, Baltimore, MA, USA.","DOI":"10.1145\/3159450.3159579"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1457394","DOI":"10.1080\/10872981.2018.1457394","article-title":"Knowledge maps: A tool for online assessment with automated feedback","volume":"23","author":"Ho","year":"2018","journal-title":"Med. Educ. Online"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1002\/sce.21504","article-title":"Automated text scoring and real-time adjustable feedback: Supporting revision of scientific arguments involving uncertainty","volume":"103","author":"Lee","year":"2019","journal-title":"Sci. Educ."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/s10664-020-09920-w","article-title":"Automated patch assessment for program repair at scale","volume":"26","author":"Ye","year":"2021","journal-title":"Empir. Softw. Eng."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Parihar, S., Dadachanji, Z., Singh, P.K., Das, R., Karkare, A., and Bhattacharya, A. (2017, January 3\u20135). Automatic Grading and Feedback Using Program Repair for Introductory Programming Courses. Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE\u201917, Bologna, Italy.","DOI":"10.1145\/3059009.3059026"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Leite, A., and Blanco, S.A. (2020, January 11\u201314). Effects of Human vs. Automatic Feedback on Students\u2019 Understanding of AI Concepts and Programming Style. Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE\u201920, Portland, OR, USA.","DOI":"10.1145\/3328778.3366921"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Pape, S., Flake, J., Beckmann, A., and J\u00fcrjens, J. (2016, January 14\u201322). STAGE: A Software Tool for Automatic Grading of Testing Exercises: Case Study Paper. Proceedings of the 38th International Conference on Software Engineering Companion, ICSE\u201916, Austin, TX, USA.","DOI":"10.1145\/2889160.2889203"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wrenn, J., and Krishnamurthi, S. (2019, January 12\u201314). Executable Examples for Programming Problem Comprehension. Proceedings of the 2019 ACM Conference on International Computing Education Research, ICER\u201919, Toronto, ON, Canada.","DOI":"10.1145\/3291279.3339416"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1504\/IJKL.2015.071054","article-title":"Teaching novice programmers using ProgTest","volume":"10","author":"Souza","year":"2015","journal-title":"Int. J. Knowl. Learn."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Smith, R., Tang, T., Warren, J., and Rixner, S. (2017, January 3\u20135). An Automated System for Interactively Learning Software Testing. Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE\u201917, Bologna, Italy.","DOI":"10.1145\/3059009.3059022"},{"key":"ref_44","unstructured":"Buffardi, K., Valdivia, P., and Rogers, D. (March, January 27). Measuring Unit Test Accuracy. Proceedings of the 50th ACM Technical Symposium on Computer Science Education, SIGCSE\u201919, Minneapolis, MN, USA."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1080\/10508406.2014.954750","article-title":"Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming","volume":"23","author":"Blikstein","year":"2014","journal-title":"J. Learn. Sci."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Tahaei, N., and Noelle, D.C. (2018, January 13\u201315). Automated Plagiarism Detection for Computer Programming Exercises Based on Patterns of Resubmission. Proceedings of the 2018 ACM Conference on International Computing Education Research, ICER\u201918, Espoo, Finland.","DOI":"10.1145\/3230977.3231006"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.cola.2019.03.006","article-title":"Code4Bench: A multidimensional benchmark of Codeforces data for different program analysis techniques","volume":"53","author":"Majd","year":"2019","journal-title":"J. Comput. Lang."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2948","DOI":"10.1007\/s10664-017-9552-y","article-title":"A Correlation Study between Automated Program Repair and Test-Suite Metrics","volume":"23","author":"Yi","year":"2018","journal-title":"Empirical Softw. Engg."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Adler, F., Fraser, G., Grundinger, E., Korber, N., Labrenz, S., Lerchenberger, J., Lukasczyk, S., and Schweikl, S. (2021, January 27\u201328). Improving Readability of Scratch Programs with Search-based Refactoring. Proceedings of the 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM), Luxembourg.","DOI":"10.1109\/SCAM52516.2021.00023"},{"key":"ref_50","unstructured":"Porfirio, A., Pereira, R., and Maschio, E. (2021). Automatic Source Code Evaluation: A Systematic Mapping, Federal University of Technology (UTFPR). Technical report."}],"container-title":["Electronics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-9292\/12\/10\/2254\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:35:37Z","timestamp":1760124937000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-9292\/12\/10\/2254"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,15]]},"references-count":50,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["electronics12102254"],"URL":"https:\/\/doi.org\/10.3390\/electronics12102254","relation":{},"ISSN":["2079-9292"],"issn-type":[{"value":"2079-9292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,15]]}}}