{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,24]],"date-time":"2025-08-24T01:16:37Z","timestamp":1755998197688,"version":"3.41.0"},"reference-count":27,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2005,9,1]],"date-time":"2005-09-01T00:00:00Z","timestamp":1125532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Educ. Resour. Comput."],"published-print":{"date-parts":[[2005,9]]},"abstract":"<jats:p>In this paper, we present our experiences in using two automatic assessment tools, TRAKLA and TRAKLA2, in a second course of programming. In this course, 500--700 students have been enrolled annually during the period 1993--2004. The tools are specifically designed for assessing algorithm simulation exercises in which students simulate the working of algorithms at a conceptual level. Both of these tools allow students to resubmit their solutions after getting feedback. However, the resubmission policy has changed considerably during the period. Those changes reflect the students performance in the exercises. We conclude that an encouraging grading policy, i.e., the more exercises they solve the better grades they achive, combined with an option to resubmit the solution is a very important factor promoting students' learning. However, in order to prevent aimless trial-and-error problem solving method, the number of resubmissions allowed per assignment should be carefully controlled.<\/jats:p>","DOI":"10.1145\/1163405.1163412","type":"journal-article","created":{"date-parts":[[2006,10,18]],"date-time":"2006-10-18T18:11:32Z","timestamp":1161195092000},"page":"7","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["Experiences on automatically assessed algorithm simulation exercises with different resubmission policies"],"prefix":"10.1145","volume":"5","author":[{"given":"Lauri","family":"Malmi","sequence":"first","affiliation":[{"name":"Helsinki University of Technology, Finland"}]},{"given":"Ville","family":"Karavirta","sequence":"additional","affiliation":[{"name":"Helsinki University of Technology, Finland"}]},{"given":"Ari","family":"Korhonen","sequence":"additional","affiliation":[{"name":"Helsinki University of Technology, Finland"}]},{"given":"Jussi","family":"Nikander","sequence":"additional","affiliation":[{"name":"Helsinki University of Technology, Finland"}]}],"member":"320","published-online":{"date-parts":[[2005,9]]},"reference":[{"volume-title":"Proceedings of the 30th SIGCSE Technical Symposium on Computer Science Education","author":"Baker R. S.","key":"e_1_2_1_1_1","unstructured":"Baker , R. S. , Boilen , M. , Goodrich , M. T. , Tamassia , R. , and Stibel , B. A . 1999. Testers and visualizers for teaching data structures . In Proceedings of the 30th SIGCSE Technical Symposium on Computer Science Education . New Orleans, LA, ACM Press, New York, 261--265. 10.1145\/299649.299779 Baker, R. S., Boilen, M., Goodrich, M. T., Tamassia, R., and Stibel, B. A. 1999. Testers and visualizers for teaching data structures. In Proceedings of the 30th SIGCSE Technical Symposium on Computer Science Education. New Orleans, LA, ACM Press, New York, 261--265. 10.1145\/299649.299779"},{"key":"e_1_2_1_2_1","first-page":"45","article-title":"Constructivism in computer science education","volume":"20","author":"Ben-Ari M.","year":"2001","unstructured":"Ben-Ari , M. 2001 . Constructivism in computer science education . Journal of Computers in Mathematics and Science Teaching 20 , 1, 45 -- 73 . Ben-Ari, M. 2001. Constructivism in computer science education. Journal of Computers in Mathematics and Science Teaching 20, 1, 45--73.","journal-title":"Journal of Computers in Mathematics and Science Teaching"},{"volume-title":"Proceedings of the 1st International Conference of Computer Based Learning","author":"Benford S.","key":"e_1_2_1_3_1","unstructured":"Benford , S. , Burke , E. , Foxley , E. , Gutteridge , N. , and Zin , A. M . 1993. Ceilidh: A course administration and marking system . In Proceedings of the 1st International Conference of Computer Based Learning . Vienna, Austria. Benford, S., Burke, E., Foxley, E., Gutteridge, N., and Zin, A. M. 1993. Ceilidh: A course administration and marking system. In Proceedings of the 1st International Conference of Computer Based Learning. Vienna, Austria."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 31st SIGCSE Technical Symposium on Computer Science Education. ACM Press","author":"Bridgeman S.","year":"1843","unstructured":"Bridgeman , S. , Goodrich , M. T. , Kobourov , S. G. , and Tamassia , R . 2000. PILOT: An interactive tool for learning and grading . In Proceedings of the 31st SIGCSE Technical Symposium on Computer Science Education. ACM Press , New York. 139--143. 10.1145\/330908.33 1843 Bridgeman, S., Goodrich, M. T., Kobourov, S. G., and Tamassia, R. 2000. PILOT: An interactive tool for learning and grading. In Proceedings of the 31st SIGCSE Technical Symposium on Computer Science Education. ACM Press, New York. 139--143. 10.1145\/330908.331843"},{"key":"e_1_2_1_5_1","unstructured":"Brown M. H. 1988. Algorithm Animation. MIT Press Cambridge MA.  Brown M. H. 1988. Algorithm Animation. MIT Press Cambridge MA."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/960492.960539"},{"volume-title":"Proceedings of Symposium on Visual Languages. Isle of Capri., Italy, IEEE, 360--367","author":"Haajanen J.","key":"e_1_2_1_7_1","unstructured":"Haajanen , J. , Pesonius , M. , Sutinen , E. , Tarhio , J. , Ter\u00e4svirta , T. , and Vanninen , P . 1997. Animation of user algorithms on the Web . In Proceedings of Symposium on Visual Languages. Isle of Capri., Italy, IEEE, 360--367 . Haajanen, J., Pesonius, M., Sutinen, E., Tarhio, J., Ter\u00e4svirta, T., and Vanninen, P. 1997. Animation of user algorithms on the Web. In Proceedings of Symposium on Visual Languages. Isle of Capri., Italy, IEEE, 360--367."},{"volume-title":"Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education. ACM Press","author":"Higgins C.","key":"e_1_2_1_8_1","unstructured":"Higgins , C. , Symeonidis , P. , and Tsintsifas , A . 2002. The marking system for CourseMaster . In Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education. ACM Press , New York, 46--50. 10.1145\/544414.544431 Higgins, C., Symeonidis, P., and Tsintsifas, A. 2002. The marking system for CourseMaster. In Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education. ACM Press, New York, 46--50. 10.1145\/544414.544431"},{"volume-title":"Proceedings of HYPERMEDIA in Vaasa. 141--147","author":"Hyv\u00f6nen J.","key":"e_1_2_1_9_1","unstructured":"Hyv\u00f6nen , J. and Malmi , L . 1993. TRAKLA---a system for teaching algorithms using email and a graphical editor . In Proceedings of HYPERMEDIA in Vaasa. 141--147 . Hyv\u00f6nen, J. and Malmi, L. 1993. TRAKLA---a system for teaching algorithms using email and a graphical editor. In Proceedings of HYPERMEDIA in Vaasa. 141--147."},{"volume-title":"Proceedings of 28th ACM SIGCSE Symposium on Computer Science Education. 335--339","author":"Jackson D.","key":"e_1_2_1_10_1","unstructured":"Jackson , D. and Usher , M . 1997. Grading student programs using ASSYST . In Proceedings of 28th ACM SIGCSE Symposium on Computer Science Education. 335--339 . 10.1145\/268084.268210 Jackson, D. and Usher, M. 1997. Grading student programs using ASSYST. In Proceedings of 28th ACM SIGCSE Symposium on Computer Science Education. 335--339. 10.1145\/268084.268210"},{"volume-title":"Proceedings of The 5th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'00","author":"Korhonen A.","key":"e_1_2_1_11_1","unstructured":"Korhonen , A. and Malmi , L . 2000. Algorithm simulation with automatic assessment . In Proceedings of The 5th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'00 . Helsinki, Finland, ACM Press, New York. 160--163. 10.1145\/343048.343157 Korhonen, A. and Malmi, L. 2000. Algorithm simulation with automatic assessment. In Proceedings of The 5th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'00. Helsinki, Finland, ACM Press, New York. 160--163. 10.1145\/343048.343157"},{"volume-title":"Proceedings of the Working Conference on Advanced Visual Interfaces","author":"Korhonen A.","key":"e_1_2_1_12_1","unstructured":"Korhonen , A. and Malmi , L . 2002. Matrix---Concept animation and algorithm simulation system . In Proceedings of the Working Conference on Advanced Visual Interfaces , Trento, Italy, ACM Press, New York. 109--114. 10.1145\/1556262.1556278 Korhonen, A. and Malmi, L. 2002. Matrix---Concept animation and algorithm simulation system. In Proceedings of the Working Conference on Advanced Visual Interfaces, Trento, Italy, ACM Press, New York. 109--114. 10.1145\/1556262.1556278"},{"volume-title":"Proceedings of the Third Program Visualization Workshop, A. Korhonen, ed","author":"Korhonen A.","key":"e_1_2_1_13_1","unstructured":"Korhonen , A. and Malmi , L . 2004. Taxonomy of visual algorithm simulation exercises . In Proceedings of the Third Program Visualization Workshop, A. Korhonen, ed . Warwick, UK, 118--125. Korhonen, A. and Malmi, L. 2004. Taxonomy of visual algorithm simulation exercises. In Proceedings of the Third Program Visualization Workshop, A. Korhonen, ed. Warwick, UK, 118--125."},{"volume-title":"Proceedings of The 7th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'02","author":"Korhonen A.","key":"e_1_2_1_14_1","unstructured":"Korhonen , A. , Malmi , L. , Myllyselk\u00e4 , P. , and Scheinin , P . 2002. Does it make a difference if students exercise on the web or in the classroom? In Proceedings of The 7th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'02 . Aarhus, Denmark, ACM Press, New York. 121--124. 10.1145\/544414.544452 Korhonen, A., Malmi, L., Myllyselk\u00e4, P., and Scheinin, P. 2002. Does it make a difference if students exercise on the web or in the classroom? In Proceedings of The 7th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'02. Aarhus, Denmark, ACM Press, New York. 121--124. 10.1145\/544414.544452"},{"volume-title":"Proceedings of Kolin Kolistelut\/Koli Calling---Third Annual Baltic Conference on Computer Science Education","author":"Korhonen A.","key":"e_1_2_1_15_1","unstructured":"Korhonen , A. , Malmi , L. , and Silvasti , P . 2003. TRAKLA2: A framework for automatically assessed visual algorithm simulation exercises . In Proceedings of Kolin Kolistelut\/Koli Calling---Third Annual Baltic Conference on Computer Science Education . Joensuu, Finland. 48--56. Korhonen, A., Malmi, L., and Silvasti, P. 2003. TRAKLA2: A framework for automatically assessed visual algorithm simulation exercises. In Proceedings of Kolin Kolistelut\/Koli Calling---Third Annual Baltic Conference on Computer Science Education. Joensuu, Finland. 48--56."},{"volume-title":"Proceedings of Kolin Kolistelut\/Koli Calling---Fourth Finnish\/Baltic Sea Conference on Computer Science Education","author":"Laakso M.-J.","key":"e_1_2_1_16_1","unstructured":"Laakso , M.-J. , Salakoski , T. , Korhonen , A. , and Malmi , L . 2004. Automatic assessment of exercises for algorithms and data structures---a case study with TRAKLA2 . In Proceedings of Kolin Kolistelut\/Koli Calling---Fourth Finnish\/Baltic Sea Conference on Computer Science Education . Helsinki University of Technology, 28--36. Laakso, M.-J., Salakoski, T., Korhonen, A., and Malmi, L. 2004. Automatic assessment of exercises for algorithms and data structures---a case study with TRAKLA2. In Proceedings of Kolin Kolistelut\/Koli Calling---Fourth Finnish\/Baltic Sea Conference on Computer Science Education. Helsinki University of Technology, 28--36."},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","first-page":"267","DOI":"10.15388\/infedu.2004.19","article-title":"Visual algorithm simulation exercise system with automatic assessment: TRAKLA2","volume":"3","author":"Malmi L.","year":"2004","unstructured":"Malmi , L. , Karavirta , V. , Korhonen , A. , Nikander , J. , Sepp\u00e4l\u00e4 , O. , and Silvasti , P. 2004 . Visual algorithm simulation exercise system with automatic assessment: TRAKLA2 . Informatics in Education 3 , 2, 267 -- 288 . Malmi, L., Karavirta, V., Korhonen, A., Nikander, J., Sepp\u00e4l\u00e4, O., and Silvasti, P. 2004. Visual algorithm simulation exercise system with automatic assessment: TRAKLA2. Informatics in Education 3, 2, 267--288.","journal-title":"Informatics in Education"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1015022619307"},{"volume-title":"Proceedings of the 15th international conference on Software Engineering. IEEE Computer Society Press. 456--465","author":"Mukherjea S.","key":"e_1_2_1_19_1","unstructured":"Mukherjea , S. and Stasko , J. T . 1993. Applying algorithm animation techniques for program tracing, debugging, and understanding . In Proceedings of the 15th international conference on Software Engineering. IEEE Computer Society Press. 456--465 . Mukherjea, S. and Stasko, J. T. 1993. Applying algorithm animation techniques for program tracing, debugging, and understanding. In Proceedings of the 15th international conference on Software Engineering. IEEE Computer Society Press. 456--465."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the SIGCSE Session","author":"Naps T. L.","year":"1829","unstructured":"Naps , T. L. , Eagan , J. R. , and Norton , L. L . 2000. JHAV\u00c9: An environment to actively engage students in web-based algorithm visualizations . In Proceedings of the SIGCSE Session . Austin, Texas. ACM Press, New York. 109--113. 10.1145\/330908.33 1829 Naps, T. L., Eagan, J. R., and Norton, L. L. 2000. JHAV\u00c9: An environment to actively engage students in web-based algorithm visualizations. In Proceedings of the SIGCSE Session. Austin, Texas. ACM Press, New York. 109--113. 10.1145\/330908.331829"},{"volume-title":"Some observations on mental models","author":"Norman D. A.","key":"e_1_2_1_21_1","unstructured":"Norman , D. A. 1983. Some observations on mental models . In Mental Models, Gentner D. and A. Stevens, eds. Lawrence Erlbaum Associates , Mahwah, NJ , 7--14. Norman, D. A. 1983. Some observations on mental models. In Mental Models, Gentner D. and A. Stevens, eds. Lawrence Erlbaum Associates, Mahwah, NJ, 7--14."},{"volume-title":"Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education","author":"Pierson W.","key":"e_1_2_1_22_1","unstructured":"Pierson , W. and Rodger , S . 1998. Web-based animation of data structures using JAWAA . In Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education . Atlanta, GA, ACM Press, New York. 267--271. 10.1145\/273133.274310 Pierson, W. and Rodger, S. 1998. Web-based animation of data structures using JAWAA. In Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education. Atlanta, GA, ACM Press, New York. 267--271. 10.1145\/273133.274310"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1006\/jvlc.2002.0239","article-title":"ANIMAL: A system for supporting multiple roles in algorithm animation","volume":"13","author":"R\u00f6ssling G.","year":"2002","unstructured":"R\u00f6ssling , G. and Freisleben , B. 2002 . ANIMAL: A system for supporting multiple roles in algorithm animation . Journal of Visual Languages and Computing 13 , 3, 341 -- 354 . R\u00f6ssling, G. and Freisleben, B. 2002. ANIMAL: A system for supporting multiple roles in algorithm animation. Journal of Visual Languages and Computing 13, 3, 341--354.","journal-title":"Journal of Visual Languages and Computing"},{"volume-title":"Proceedings of The 6th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'01","author":"Saikkonen R.","key":"e_1_2_1_24_1","unstructured":"Saikkonen , R. , Malmi , L. , and Korhonen , A . 2001. Fully automatic assessment of programming exercises . In Proceedings of The 6th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'01 . Canterbury, UK, ACM Press, New York. 133--136. 10.1145\/377435.377666 Saikkonen, R., Malmi, L., and Korhonen, A. 2001. Fully automatic assessment of programming exercises. In Proceedings of The 6th Annual SIGCSE\/SIGCUE Conference on Innovation and Technology in Computer Science Education, ITiCSE'01. Canterbury, UK, ACM Press, New York. 133--136. 10.1145\/377435.377666"},{"volume-title":"Proceedings of the IASTED International Conference on Web-Based Education. IASTED","author":"Silvasti P.","key":"e_1_2_1_25_1","unstructured":"Silvasti , P. , Malmi , L. , and Torvinen , P . 2004. Collecting statistical data of the usage of a web-based educational software . In Proceedings of the IASTED International Conference on Web-Based Education. IASTED , Innsbruck, Austria, 107--110. Silvasti, P., Malmi, L., and Torvinen, P. 2004. Collecting statistical data of the usage of a web-based educational software. In Proceedings of the IASTED International Conference on Web-Based Education. IASTED, Innsbruck, Austria, 107--110."},{"key":"e_1_2_1_26_1","volume-title":"The Proceedings of the 28th SIGCSE Technical Symposium on Computer Science Education","author":"Stasko J. T.","year":"1997","unstructured":"Stasko , J. T. 1997 . Using student-built algorithm animations as learning aids . In The Proceedings of the 28th SIGCSE Technical Symposium on Computer Science Education . San Jose, CA, ACM Press, New York. 25--29. 10.1145\/268084.268091 Stasko, J. T. 1997. Using student-built algorithm animations as learning aids. In The Proceedings of the 28th SIGCSE Technical Symposium on Computer Science Education. San Jose, CA, ACM Press, New York. 25--29. 10.1145\/268084.268091"},{"volume-title":"The Proceedings of Int. Symposium on Technologies of Information and Communication in Education for Engineering and Industry (TICE2002)","author":"Vihtonen E.","key":"e_1_2_1_28_1","unstructured":"Vihtonen , E. and Ageenko , E . 2002. Viope-computer supported environment for learning programming languages . In The Proceedings of Int. Symposium on Technologies of Information and Communication in Education for Engineering and Industry (TICE2002) . Lyon, France. 371--372. Vihtonen, E. and Ageenko, E. 2002. Viope-computer supported environment for learning programming languages. In The Proceedings of Int. Symposium on Technologies of Information and Communication in Education for Engineering and Industry (TICE2002). Lyon, France. 371--372."}],"container-title":["Journal on Educational Resources in Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1163405.1163412","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1163405.1163412","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T16:19:02Z","timestamp":1750263542000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1163405.1163412"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2005,9]]},"references-count":27,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2005,9]]}},"alternative-id":["10.1145\/1163405.1163412"],"URL":"https:\/\/doi.org\/10.1145\/1163405.1163412","relation":{},"ISSN":["1531-4278","1531-4278"],"issn-type":[{"type":"print","value":"1531-4278"},{"type":"electronic","value":"1531-4278"}],"subject":[],"published":{"date-parts":[[2005,9]]},"assertion":[{"value":"2005-09-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}