{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T22:38:27Z","timestamp":1770676707704,"version":"3.49.0"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2005,9,1]],"date-time":"2005-09-01T00:00:00Z","timestamp":1125532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Educ. Resour. Comput."],"published-print":{"date-parts":[[2005,9]]},"abstract":"<jats:p>Computer programming lends itself to automated assessment. With appropriate software tools, program correctness can be measured, along with an indication of quality according to a set of metrics. Furthermore, the regularity of program code allows plagiarism detection to be an integral part of the tools that support assessment. In this paper, we describe a submission and assessment system, called BOSS, that supports coursework assessment through collecting submissions, performing automatic tests for correctness and quality, checking for plagiarism, and providing an interface for marking and delivering feedback. We describe how automated assessment is incorporated into BOSS such that it supports, rather than constrains, assessment. The pedagogic and administrative issues that are affected by the assessment process are also discussed.<\/jats:p>","DOI":"10.1145\/1163405.1163407","type":"journal-article","created":{"date-parts":[[2006,10,18]],"date-time":"2006-10-18T18:11:32Z","timestamp":1161195092000},"page":"2","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":138,"title":["The boss online submission and assessment system"],"prefix":"10.1145","volume":"5","author":[{"given":"Mike","family":"Joy","sequence":"first","affiliation":[{"name":"University of Warwick, United Kingdom"}]},{"given":"Nathan","family":"Griffiths","sequence":"additional","affiliation":[{"name":"University of Warwick, United Kingdom"}]},{"given":"Russell","family":"Boyatt","sequence":"additional","affiliation":[{"name":"University of Warwick, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2005,9]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"HERDSA 2003","author":"Bancroft P.","year":"2004","unstructured":"Bancroft , P. , Hynd , J. , Santo , F. D. , and Reye , J . 2003. Web-based assignment submission and electronic marking . In HERDSA 2003 . IEEE. Available : http:\/\/surveys.canterbury. ac.nz\/herdsa03\/pdfsref\/Y1007.pdf (accessed: 30 May , 2004 ). Bancroft, P., Hynd, J., Santo, F. D., and Reye, J. 2003. Web-based assignment submission and electronic marking. In HERDSA 2003. IEEE. Available: http:\/\/surveys.canterbury. ac.nz\/herdsa03\/pdfsref\/Y1007.pdf (accessed: 30 May, 2004)."},{"key":"e_1_2_1_2_1","unstructured":"Bloom B. S. and Krathwohl D. R. 1956. Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive Domain. Longman London.  Bloom B. S. and Krathwohl D. R. 1956. Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive Domain. Longman London."},{"key":"e_1_2_1_3_1","volume-title":"ITTC","author":"Blumenstein M.","year":"2004","unstructured":"Blumenstein , M. , Green , S. , Nguyen , A. , and Muthukkumarasamy , V . 2004. Game: A generic automated marking environment for programming assessment . In ITTC 2004 . IEEE, 212--216. Blumenstein, M., Green, S., Nguyen, A., and Muthukkumarasamy, V. 2004. Game: A generic automated marking environment for programming assessment. In ITTC 2004. IEEE, 212--216."},{"key":"e_1_2_1_4_1","volume-title":"The BOSS online submission system. online. Available: http:\/\/boss.org.uk\/ (accessed","year":"2004","unstructured":"BOSS. 2004. The BOSS online submission system. online. Available: http:\/\/boss.org.uk\/ (accessed 19 December , 2004 ). BOSS. 2004. The BOSS online submission system. online. Available: http:\/\/boss.org.uk\/ (accessed 19 December, 2004)."},{"key":"e_1_2_1_5_1","volume-title":"Blueprint for Computer-Assisted Assessment","author":"Bull J.","unstructured":"Bull , J. and McKenna , C. 2001. Blueprint for Computer-Assisted Assessment . CAA Centre , University of Loughborough. Bull, J. and McKenna, C. 2001. Blueprint for Computer-Assisted Assessment. CAA Centre, University of Loughborough."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/782941.782993"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/274790.273178"},{"key":"e_1_2_1_8_1","volume-title":"Copycatch gold. online. Available: http:\/\/www.copycatchgold. com\/ (accessed","author":"Software Development","year":"2004","unstructured":"CFL Software Development . 2004. Copycatch gold. online. Available: http:\/\/www.copycatchgold. com\/ (accessed : 30 March , 2004 ). CFL Software Development. 2004. Copycatch gold. online. Available: http:\/\/www.copycatchgold. com\/ (accessed: 30 March, 2004)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(03)00030-7"},{"key":"e_1_2_1_10_1","volume-title":"online. Available: http:\/\/www.derby.ac.uk\/assess\/ (accessed","author":"TRIADS.","year":"2004","unstructured":"CIAD. 2004. TRIADS. online. Available: http:\/\/www.derby.ac.uk\/assess\/ (accessed : 25 April , 2004 ) CIAD. 2004. TRIADS. online. Available: http:\/\/www.derby.ac.uk\/assess\/ (accessed: 25 April, 2004)"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/216511.216539"},{"key":"e_1_2_1_12_1","volume-title":"Promoting Deep Learning through Assessment and Teaching","author":"Entwistle N.","unstructured":"Entwistle , N. 2001. Promoting Deep Learning through Assessment and Teaching . AAHE , Washington, DC . Entwistle, N. 2001. Promoting Deep Learning through Assessment and Teaching. AAHE, Washington, DC."},{"key":"e_1_2_1_13_1","volume-title":"GNU general public license. online. Available: http:\/\/www.gnu. org\/copyleft\/gpl.html\/ (accessed","author":"Free Software Foundation","year":"2004","unstructured":"Free Software Foundation . 2004. GNU general public license. online. Available: http:\/\/www.gnu. org\/copyleft\/gpl.html\/ (accessed : 25 April , 2004 ). Free Software Foundation. 2004. GNU general public license. online. Available: http:\/\/www.gnu. org\/copyleft\/gpl.html\/ (accessed: 25 April, 2004)."},{"key":"e_1_2_1_14_1","volume-title":"ICITA","author":"Ghosh M.","year":"2002","unstructured":"Ghosh , M. , Verma , B. , and Nguyen , A . 2002. An automatic assessment marking and plagiarism detection . In ICITA 2002 . IEEE. Ghosh, M., Verma, B., and Nguyen, A. 2002. An automatic assessment marking and plagiarism detection. In ICITA 2002. IEEE."},{"key":"e_1_2_1_15_1","volume-title":"Tech. Rep. CS-RR-415, Department of Computer Science","author":"Heng P.","year":"2005","unstructured":"Heng , P. , Joy , M. , Boyatt , R. , and Griffiths , N . 2005 . Evaluation of the BOSS online submission and assessment system. Tech. Rep. CS-RR-415, Department of Computer Science , University of Warwick Coventry, UK. Heng, P., Joy, M., Boyatt, R., and Griffiths, N. 2005. Evaluation of the BOSS online submission and assessment system. Tech. Rep. CS-RR-415, Department of Computer Science, University of Warwick Coventry, UK."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026364126982"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the First Australasian Conference on Computer Science Education. ACM","author":"Hurst A.","year":"1996","unstructured":"Hurst , A. 1996 . Literate programming as an aid to marking student assignments . In Proceedings of the First Australasian Conference on Computer Science Education. ACM , New York. 280--286. Available : http:\/\/www.literateprogramming.com\/lpin-assess.pdf (accessed: 30 March, 2004). 10.1145\/369585.369650 Hurst, A. 1996. Literate programming as an aid to marking student assignments. In Proceedings of the First Australasian Conference on Computer Science Education. ACM, New York. 280--286. Available: http:\/\/www.literateprogramming.com\/lpin-assess.pdf (accessed: 30 March, 2004). 10.1145\/369585.369650"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/65738.65741"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/13.762946"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 3rd Annual Conference of the LTSN Centre for Information and Computer Sciences. LTSN Centre for Information and Computer Sciences. 31--36","author":"Joy M.","unstructured":"Joy , M. , Griffiths , N. , Stott , M. , Harley , J. , Wattebot , C. , and Holt , D . 2002. Coresoft: A framework for student data . In Proceedings of the 3rd Annual Conference of the LTSN Centre for Information and Computer Sciences. LTSN Centre for Information and Computer Sciences. 31--36 . Joy, M., Griffiths, N., Stott, M., Harley, J., Wattebot, C., and Holt, D. 2002. Coresoft: A framework for student data. In Proceedings of the 3rd Annual Conference of the LTSN Centre for Information and Computer Sciences. LTSN Centre for Information and Computer Sciences. 31--36."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/27.2.97"},{"key":"e_1_2_1_22_1","volume-title":"JUnit: The Definitive Guide. O'Reilly","author":"Lane D.","unstructured":"Lane , D. 2004. JUnit: The Definitive Guide. O'Reilly , Sebastopol, CA . Lane, D. 2004. JUnit: The Definitive Guide. O'Reilly, Sebastopol, CA."},{"key":"e_1_2_1_23_1","volume-title":"The CASTLE toolkit. online. Available: http:\/\/www.le.ac.uk\/castle\/ (accessed","author":"Leicester University","year":"2004","unstructured":"Leicester University . 2004. The CASTLE toolkit. online. Available: http:\/\/www.le.ac.uk\/castle\/ (accessed : 25 April , 2004 ). Leicester University. 2004. The CASTLE toolkit. online. Available: http:\/\/www.le.ac.uk\/castle\/ (accessed: 25 April, 2004)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-024X(19990710)29:8%3C721::AID-SPE257%3E3.3.CO;2-S"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/271125.271155"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/BF00203968","article-title":"A fast software one way hash function","volume":"3","author":"Merkle R.","year":"1990","unstructured":"Merkle , R. 1990 . A fast software one way hash function . Journal of Cryptology 3 , 1, 43 -- 58 . Merkle, R. 1990. A fast software one way hash function. Journal of Cryptology 3, 1, 43--58.","journal-title":"Journal of Cryptology"},{"key":"e_1_2_1_27_1","volume-title":"Usability guidelines. Online. Available: http:\/\/www.mit.edu\/ist\/usability\/usability-guidelines.html (accessed","author":"MIT Usability Group","year":"2004","unstructured":"MIT Usability Group . 2005. Usability guidelines. Online. Available: http:\/\/www.mit.edu\/ist\/usability\/usability-guidelines.html (accessed : 30 September , 2004 ). MIT Usability Group. 2005. Usability guidelines. Online. Available: http:\/\/www.mit.edu\/ist\/usability\/usability-guidelines.html (accessed: 30 September, 2004)."},{"key":"e_1_2_1_28_1","volume-title":"useit.com. Online. Available: http:\/\/www.useit.com\/ (accessed","author":"Nielsen J.","year":"2004","unstructured":"Nielsen , J. 2005. useit.com. Online. Available: http:\/\/www.useit.com\/ (accessed : 30 September , 2004 ). Nielsen, J. 2005. useit.com. Online. Available: http:\/\/www.useit.com\/ (accessed: 30 September, 2004)."},{"key":"e_1_2_1_29_1","volume-title":"online. Available: http:\/\/perception. questionmark.com\/ (accessed","author":"Questionmark","year":"2004","unstructured":"Questionmark . 2004. Questionmark Perception. online. Available: http:\/\/perception. questionmark.com\/ (accessed : 30 March , 2004 ). Questionmark. 2004. Questionmark Perception. online. Available: http:\/\/perception. questionmark.com\/ (accessed: 30 March, 2004)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/65294.71198"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Saikkonen R. Malmi L. and Korhonen A. 2003. Fully automatic assessment of programming exercises. In ITiCSE 2001. ACM 133--136. 10.1145\/377435.377666   Saikkonen R. Malmi L. and Korhonen A. 2003. Fully automatic assessment of programming exercises. In ITiCSE 2001. ACM 133--136. 10.1145\/377435.377666","DOI":"10.1145\/507758.377666"},{"key":"e_1_2_1_32_1","volume-title":"Designing the User Interface","author":"Shneiderman B.","unstructured":"Shneiderman , B. 1998. Designing the User Interface , ( 3 rd ed.). Addison-Wesley , Reading, MA . Shneiderman, B. 1998. Designing the User Interface, (3rd ed.). Addison-Wesley, Reading, MA.","edition":"3"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 9th Annual Conference on the Innovation and Technology in Computer Science Education (ITiCSE","author":"Sitthiworachart J.","year":"2004","unstructured":"Sitthiworachart , J. and Joy , M . 2004. Effective peer assessment for learning computer programming . In Proceedings of the 9th Annual Conference on the Innovation and Technology in Computer Science Education (ITiCSE 2004 ). 122--126. 10.1145\/1007996.1008030 Sitthiworachart, J. and Joy, M. 2004. Effective peer assessment for learning computer programming. In Proceedings of the 9th Annual Conference on the Innovation and Technology in Computer Science Education (ITiCSE 2004). 122--126. 10.1145\/1007996.1008030"},{"key":"e_1_2_1_34_1","volume-title":"online. Available: http:\/\/www.webct.com\/ (accessed","author":"Web CT.","year":"2004","unstructured":"Web CT. 2004. Web CT. online. Available: http:\/\/www.webct.com\/ (accessed : 30 March , 2004 ). WebCT. 2004. WebCT. online. Available: http:\/\/www.webct.com\/ (accessed: 30 March, 2004)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1086339.1086341"},{"key":"e_1_2_1_36_1","volume-title":"EISTA 2004. 1861","author":"Yau J.","year":"1865","unstructured":"Yau , J. and Joy , M . 2004. Introducing Java: A case for fundamentals-first . In EISTA 2004. 1861 -- 1865 . Yau, J. and Joy, M. 2004. Introducing Java: A case for fundamentals-first. In EISTA 2004. 1861--1865."}],"container-title":["Journal on Educational Resources in Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1163405.1163407","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1163405.1163407","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T16:19:02Z","timestamp":1750263542000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1163405.1163407"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2005,9]]},"references-count":36,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2005,9]]}},"alternative-id":["10.1145\/1163405.1163407"],"URL":"https:\/\/doi.org\/10.1145\/1163405.1163407","relation":{},"ISSN":["1531-4278","1531-4278"],"issn-type":[{"value":"1531-4278","type":"print"},{"value":"1531-4278","type":"electronic"}],"subject":[],"published":{"date-parts":[[2005,9]]},"assertion":[{"value":"2005-09-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}