{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T20:52:18Z","timestamp":1773694338833,"version":"3.50.1"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2005,9,1]],"date-time":"2005-09-01T00:00:00Z","timestamp":1125532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Educ. Resour. Comput."],"published-print":{"date-parts":[[2005,9]]},"abstract":"<jats:p>Systems that automatically assess student programming assignments have been designed and used for over forty years. Systems that objectively test and mark student programming work were developed simultaneously with programming assessment in the computer science curriculum. This article reviews a number of influential automatic assessment systems, including descriptions of the earliest systems, and presents some of the most recent developments. The final sections explore a number of directions automated assessment systems may take, presenting current developments alongside a number of important emerging e-learning specifications.<\/jats:p>","DOI":"10.1145\/1163405.1163409","type":"journal-article","created":{"date-parts":[[2006,10,18]],"date-time":"2006-10-18T18:11:32Z","timestamp":1161195092000},"page":"4","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":263,"title":["Automatic test-based assessment of programming"],"prefix":"10.1145","volume":"5","author":[{"given":"Christopher","family":"Douce","sequence":"first","affiliation":[{"name":"Feedback Instruments, Crowborough, UK"}]},{"given":"David","family":"Livingstone","sequence":"additional","affiliation":[{"name":"Kingston University, Kingston-upon-Thames, UK"}]},{"given":"James","family":"Orwell","sequence":"additional","affiliation":[{"name":"Kingston University, Kingston-upon-Thames, UK"}]}],"member":"320","published-online":{"date-parts":[[2005,9]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Beck K. 2003. Test Driven Development: By Example. Addison-Wesley Boston MA.   Beck K. 2003. Test Driven Development: By Example. Addison-Wesley Boston MA."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education. 10","author":"Daly C."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education. 210--213","author":"Daly C."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 4th Annual SIGCSE\/SIGCUE ITiCSE Conference on Innovation and Technology in Computer Science Education. 155--158","author":"Daly C.","year":"1999"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the International Conference on Education and Information Systems: Technologies and Applications. 421--426","author":"Edwards S. H.","year":"2003"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the OOPSLA'03 Conference. Poster presentation. 318--319","author":"Edwards S. H.","year":"2003"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education. 131--141","author":"English J.","year":"2004"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"English J. 2002. Experience with a computer-assisted formal programming examination. ACM SIGCSE Bull. 34 3: Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education. 51--54. 10.1145\/544414.544432   English J. 2002. Experience with a computer-assisted formal programming examination. ACM SIGCSE Bull. 34 3: Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education. 51--54. 10.1145\/544414.544432","DOI":"10.1145\/637610.544432"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 5th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education. 168--171","author":"English J."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/364914.364937"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 32nd SIGCSE Technical Symposium on Computer Science Education. 139--143","author":"Gehringer E. F.","year":"2001"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/362946.362981"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026364126982"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/367415.367422"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/0360-1315(93)90086-X"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/65738.65741"},{"key":"e_1_2_1_17_1","volume-title":"Technical Symposium on Computer Science Education, Proceedings of the 28th SIGCSE","author":"Jackson D."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 5th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, 164--167","author":"Jackson D.","year":"2000"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 6th Annual Conference on the Teaching of Computing","author":"Joy M."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 5th Annual SIGCSE\/SIGCUE ItiCSE Conference on Innovation and Technology in Computer Science Education. 160--163","author":"Korhonen A."},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Korhonen A. Malmi L. Myllyselk P. and Scheinin P. 2002. Does it make a difference if students exercise on the web or in the classroom&quest; In Proceedings of the 7th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education ITiCSE02 (Aarhus Denmark) ACM Press New York 121--124. 10.1145\/544414.544452   Korhonen A. Malmi L. Myllyselk P. and Scheinin P. 2002. Does it make a difference if students exercise on the web or in the classroom&quest; In Proceedings of the 7th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education ITiCSE02 (Aarhus Denmark) ACM Press New York 121--124. 10.1145\/544414.544452","DOI":"10.1145\/544414.544452"},{"key":"e_1_2_1_22_1","volume-title":"Tech. Rep. DCC-98-4, Dep. di Ci\u00eancia de Computadores, Universidade do Porto, Portugal.","author":"Leal J. P.","year":"1998"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-024X(19990710)29:8%3C721::AID-SPE257%3E3.3.CO;2-S"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1015022619307"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/BF01956028","article-title":"Automatic grading of students' ALGOL programming","volume":"4","author":"Naur P.","year":"1964","journal-title":"BIT"},{"key":"e_1_2_1_26_1","volume-title":"Computer Based Assessment Volume 2: Case Studies in Science and Computing","author":"Oliver R. G."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/65294.71198"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the ITiCSE 2001Conference, ACM Press","author":"Saikkonen R."},{"key":"e_1_2_1_29_1","unstructured":"Smythe C. etal 2005. IMS question and test interoperability Ver. 2.0 IMS Global Learning Consortium.  Smythe C. et al. 2005. IMS question and test interoperability Ver. 2.0 IMS Global Learning Consortium."},{"key":"e_1_2_1_30_1","first-page":"35","article-title":"A prototype for an intelligent tutoring system for students learning to program","volume":"1","author":"Sykes E. R.","year":"2004","journal-title":"Java. Int. J. Comput.Appl."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 4th Annual Conference on the Teaching of Computing (Centre for Teaching Computing","author":"Thoburn G."},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","DOI":"10.1145\/182107.182101","volume-title":"Kassandra: The automatic grading system. Tech. Rep.UMIACS-TR-94-59","author":"Von Matt U.","year":"1994"},{"key":"e_1_2_1_33_1","unstructured":"Wilson S. Blinco K. and Rehak D. 2004. Service-oriented frameworks: Modelling the infrastructure for the next generation of e-learning systems. JISC-CETIS. Available from JISC.  Wilson S. Blinco K. and Rehak D. 2004. Service-oriented frameworks: Modelling the infrastructure for the next generation of e-learning systems. JISC-CETIS. Available from JISC."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education. 137--141","author":"Woit D.","year":"1892"}],"container-title":["Journal on Educational Resources in Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1163405.1163409","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1163405.1163409","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T16:19:02Z","timestamp":1750263542000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1163405.1163409"}},"subtitle":["A review"],"short-title":[],"issued":{"date-parts":[[2005,9]]},"references-count":34,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2005,9]]}},"alternative-id":["10.1145\/1163405.1163409"],"URL":"https:\/\/doi.org\/10.1145\/1163405.1163409","relation":{},"ISSN":["1531-4278","1531-4278"],"issn-type":[{"value":"1531-4278","type":"print"},{"value":"1531-4278","type":"electronic"}],"subject":[],"published":{"date-parts":[[2005,9]]},"assertion":[{"value":"2005-09-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}