{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T21:05:33Z","timestamp":1793826333550,"version":"4.6.1"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"4","funder":[{"name":"King\u2019s College London College"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor, such as the module\/class leader, larger cohorts are often assessed by multiple assessors, typically teaching assistants, which increases the risk of inconsistent grading.<\/jats:p>\n          <jats:p>To investigate the consistency of human grading of programming assignments, we asked 28 participants to each grade 40 CS1 introductory Java assignments, providing grades and feedback for correctness, code elegance, readability and documentation; the 40 assignments were split into two batches of 20. The 28 participants were divided into seven groups of four (where each group graded the same 40 assignments) to allow us to investigate the consistency of a group of assessors. In the second batch of 20, we duplicated one assignment from the first to analyse the internal consistency of individual assessors.<\/jats:p>\n          <jats:p>\n            We measured the inter-rater reliability of the groups using Krippendorff\u2019s\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\alpha\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            \u2014an\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\alpha &gt; 0.667\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            is recommended to make tentative conclusions based on the rating. Our groups were inconsistent, with an average\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\alpha=0.2\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            when grading correctness and an average\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\alpha &lt; 0.1\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            for code elegance, readability and documentation.\n          <\/jats:p>\n          <jats:p>To measure the individual consistency of graders, we measured the distance between the grades they awarded for the duplicated assignment in batch one and batch two. Only one participant of the 22 who didn\u2019t notice that the assignment was a duplicate was awarded the same grade for correctness, code elegance, readability and documentation. The average grade difference was 1.79 for correctness and less than 1.6 for code elegance, readability and documentation.<\/jats:p>\n          <jats:p>Our results show that human graders in our study cannot agree on the grade to give a piece of student work and are often individually inconsistent, suggesting that the idea of a \u2018gold standard\u2019 of human grading might be flawed. This highlights that a shared rubric alone is not enough to ensure consistency, and other aspects such as assessor training and alternative grading practices should be explored to improve the consistency of human grading further when grading programming assignments.<\/jats:p>","DOI":"10.1145\/3759256","type":"journal-article","created":{"date-parts":[[2025,8,6]],"date-time":"2025-08-06T15:16:33Z","timestamp":1754493393000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["How Consistent Are Humans When Grading Programming Assignments?"],"prefix":"10.1145","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5915-9153","authenticated-orcid":false,"given":"Marcus","family":"Messer","sequence":"first","affiliation":[{"name":"Informatics, King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6086-2479","authenticated-orcid":false,"given":"Neil\u00a0C. C.","family":"Brown","sequence":"additional","affiliation":[{"name":"King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0544-2003","authenticated-orcid":false,"given":"Michael","family":"K\u00f6lling","sequence":"additional","affiliation":[{"name":"Department of Informatics, King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4933-0073","authenticated-orcid":false,"given":"Miaojing","family":"Shi","sequence":"additional","affiliation":[{"name":"College of Electronic and Information Engineering, Tongji University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,18]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1352135.1352162"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.3200\/CTCH.53.1.27-31"},{"key":"e_1_3_4_4_2","unstructured":"Marcus Messer Neil Brown Michael K\u00f6lling and Miaojing Shi. 2024. How Consistent Are Human Graders? Pre-registration. Retrieved from https:\/\/osf.io\/3nc7p\/?view_only=d37833bc919d4a1f96d0fa0dcd43b5c7"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2094131.2094152"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","unstructured":"David Azcona and Alan Smeaton. 2020. +5 Million Python & Bash Programming Submissions for 5 Courses & Grades for Computer-Based Exams over 3 academic years. DOI: 10.6084\/m9.figshare.12610958.v1","DOI":"10.6084\/m9.figshare.12610958.v1"},{"key":"e_1_3_4_7_2","volume-title":"Objects First with Java: A Practical Introduction Using BlueJ","author":"Barnes David John","year":"2006","unstructured":"David John Barnes and Michael K\u00f6lling. 2006. Objects First with Java: A Practical Introduction Using BlueJ. Pearson\/Prentice Hall, London, United Kingdom."},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2015.1024607"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3568812.3603485"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3174781.3174785"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544547"},{"key":"e_1_3_4_12_2","doi-asserted-by":"crossref","first-page":"22","DOI":"10.3389\/feduc.2018.00022","article-title":"Appropriate criteria: Key to effective rubrics","volume":"3","author":"Brookhart Susan M.","year":"2018","unstructured":"Susan M. Brookhart. 2018. Appropriate criteria: Key to effective rubrics. Frontiers in Education 3 (2018), 22.","journal-title":"Frontiers in Education"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538924"},{"key":"e_1_3_4_14_2","unstructured":"Santiago Castro. 2017. Fast Krippendorff: Fast computation of Krippendorff\u2019s alpha agreement measure. Retrieved from https:\/\/github.com\/pln-fing-udelar\/fast-krippendorff"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tsc.2014.02.003"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.NEDT.2018.06.022"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569822"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626252.3630953"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3636243.3636256"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.nepr.2018.05.013"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3430895.3460165"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2632320.2632346"},{"key":"e_1_3_4_23_2","volume-title":"A Manual for Repertory Grid Technique","author":"Fransella Fay","year":"2004","unstructured":"Fay Fransella, Richard Bell, and Don Bannister. 2004. A Manual for Repertory Grid Technique. John Wiley & Sons."},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734412"},{"issue":"2","key":"e_1_3_4_25_2","first-page":"276","article-title":"Formative plus summative assessment in large undergraduate courses: Why both","volume":"26","author":"Glazer Nirit","year":"2014","unstructured":"Nirit Glazer. 2014. Formative plus summative assessment in large undergraduate courses: Why both? International Journal of Teaching and Learning in Higher Education 26, 2 (2014), 276\u2013286.","journal-title":"International Journal of Teaching and Learning in Higher Education"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1348\/000711006X126600"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1119\/1.1634963"},{"key":"e_1_3_4_28_2","volume-title":"2017 ASEE Annual Conference & Exposition","author":"Hicks Nathan M.","year":"2017","unstructured":"Nathan M. Hicks and Heidi A. Diefes-Dux. 2017. Grader consistency in using standards-based rubrics. In Proceedings of the 2017 ASEE Annual Conference & Exposition."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1080\/0969594X.2021.1884041"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.edurev.2007.05.002"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","unstructured":"David Kane James Williams and Gillian Cappuccini-Ansfield. 2008. Student Satisfaction Surveys: The Value in Taking an Historical Perspective. 2 (Jul. 2008) 135\u2013155. DOI: 10.1080\/13538320802278347","DOI":"10.1080\/13538320802278347"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3478431.3499284"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649165.3703621"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3636243.3636258"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1080\/03075070220000662"},{"key":"e_1_3_4_36_2","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Klaus Krippendorff","year":"2013","unstructured":"Klaus Krippendorff. 2013. Content Analysis: An Introduction to Its Methodology (3rd ed.). Sage Publications, Los Angeles.","edition":"3"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3631802.3631804"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432370"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICAIBD.2018.8396196"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00047"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3017680.3017764"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-6-42"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3636515"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3372630"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339422"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/WPC.2004.1311066"},{"key":"e_1_3_4_47_2","first-page":"53","article-title":"Generic assessment rubrics for computer programming courses","volume":"15","author":"Mustapha Aida","year":"2016","unstructured":"Aida Mustapha, Noor Azah Samsudin, Nurieze Arbaiy, Rozlini Mohammed, and Isredza Rahmi Hamid. 2016. Generic assessment rubrics for computer programming courses. Turkish Online Journal of Educational Technology - TOJET 15, 1 (Jan. 2016), 53\u201368.","journal-title":"Turkish Online Journal of Educational Technology - TOJET"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602930600760868"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3517133"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.14738\/aivp.112.14406"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3519312"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602930902862859"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tsc.2017.11.004"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2018.8659243"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3517134"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3430665.3456304"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2023.2203883"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3300115.3309532"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3051466"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/6592.6594"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2999541.2999555"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474623"},{"key":"e_1_3_4_63_2","volume-title":"The Global Achievement Gap: Why Even Our Best Schools Don\u2019t Teach the New Survival Skills Our Children Need-and What We Can Do about It","author":"Wagner Tony","year":"2010","unstructured":"Tony Wagner. 2010. The Global Achievement Gap: Why Even Our Best Schools Don\u2019t Teach the New Survival Skills Our Children Need-and What We Can Do about It. Retrieved from ReadHowYouWant.com"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1080\/0309877X.2018.1499883"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2010.5673351"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-13-61"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.1977.tb00031.x"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3759256","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,18]],"date-time":"2025-09-18T17:18:20Z","timestamp":1758215900000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3759256"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,18]]},"references-count":66,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3759256"],"URL":"https:\/\/doi.org\/10.1145\/3759256","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,18]]},"assertion":[{"value":"2024-08-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-31","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}