{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T18:31:03Z","timestamp":1763663463324,"version":"3.41.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2016,6,8]],"date-time":"2016-06-08T00:00:00Z","timestamp":1465344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2016,6,27]]},"abstract":"<jats:p>The past few years has seen a proliferation of novice programming tools. The availability of a large number of systems has made it difficult for many users to choose among them. Even for education researchers, comparing the relative quality of these tools, or judging their respective suitability for a given context, is hard in many instances. For designers of such systems, assessing the respective quality of competing design decisions can be equally difficult.<\/jats:p>\n          <jats:p>Heuristic evaluation provides a practical method of assessing the quality of alternatives in these situations and of identifying potential problems with existing systems for a given target group or context. Existing sets of heuristics, however, are not specific to the domain of novice programming and thus do not evaluate all aspects of interest to us in this specialised application domain.<\/jats:p>\n          <jats:p>In this article, we propose a set of heuristics to be used in heuristic evaluations of novice programming systems. These heuristics have the potential to allow a useful assessment of the quality of a given system with lower cost than full formal user studies and greater precision than the use of existing sets of heuristics. The heuristics are described and discussed in detail. We present an evaluation of the effectiveness of the heuristics that suggests that the new set of heuristics provides additional useful information to designers not obtained with existing heuristics sets.<\/jats:p>","DOI":"10.1145\/2872521","type":"journal-article","created":{"date-parts":[[2016,6,10]],"date-time":"2016-06-10T12:46:01Z","timestamp":1465562761000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Heuristic Evaluation for Novice Programming Systems"],"prefix":"10.1145","volume":"16","author":[{"given":"Michael","family":"K\u00f6lling","sequence":"first","affiliation":[{"name":"University of Kent, Kent, UK"}]},{"given":"Fraser","family":"McKay","sequence":"additional","affiliation":[{"name":"University of Kent, Kent, UK"}]}],"member":"320","published-online":{"date-parts":[[2016,6,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/563517.563395"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447319509526114"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"A. Begel and E. Klopfer. 2005. StarLogo: A programmable complex systems modeling environment for students and teachers. Artificial Life Models in Software. Springer London 187--209.  A. Begel and E. Klopfer. 2005. StarLogo: A programmable complex systems modeling environment for students and teachers. Artificial Life Models in Software. Springer London 187--209.","DOI":"10.1007\/1-84628-214-4_8"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-007-0068-x"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/273133.274308"},{"volume-title":"Proceedings of the Meeting of the American Educational Research Association.","author":"Berland M.","key":"e_1_2_1_6_1"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2005.15"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/832277.834337"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/507758.377440"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1810543.1810611"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/359369.359375"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/611892.611966"},{"key":"e_1_2_1_13_1","unstructured":"A. Cooper R. Reimann and D. Cronin (Eds.). 2007. Errors alerts and confirmation. In About Face 3: The Essentials of Interaction Design. Wiley Indianapolis 529--550.  A. Cooper R. Reimann and D. Cronin (Eds.). 2007. Errors alerts and confirmation. In About Face 3: The Essentials of Interaction Design. Wiley Indianapolis 529--550."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"M. de Raadt R. Watson and M. Toleman. 2002. Language trends in introductory programming courses. In Informing Science: Where Parallels Intersect. 329--337.  M. de Raadt R. Watson and M. Toleman. 2002. Language trends in introductory programming courses. In Informing Science: Where Parallels Intersect. 329--337.","DOI":"10.28945\/2464"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1999747.1999807"},{"volume-title":"People and Computers V: Proceedings of the 5th Conference of the British Computer Society Human-Computer Interaction Specialist Group. 443--460","year":"1989","author":"Green T. R. G.","key":"e_1_2_1_16_1"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1006\/jvlc.1996.0009"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327590IJHC1304_03"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvlc.2006.09.001"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1151588.1151600"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124830"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1272516.1272540"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1056808.1056965"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124831"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2004.47"},{"key":"e_1_2_1_26_1","unstructured":"M. K\u00f6lling. 1999. The Design of an Object-Oriented Environment and Language for Teaching. Ph.D. Dissertation. University of Sydney Basser Department of Computer Science.  M. K\u00f6lling. 1999. The Design of an Object-Oriented Environment and Language for Teaching. Ph.D. Dissertation. University of Sydney Basser Department of Computer Science."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1868358.1868361"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1076\/csed.13.4.249.17496"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1721933.1721952"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1536513.1536516"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/800088.802839"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1868358.1868363"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1352135.1352260"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953308"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/2377916.2377967"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/182.358437"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1709424.1709450"},{"volume-title":"Retrieved","year":"2005","author":"Nielsen J.","key":"e_1_2_1_38_1"},{"key":"e_1_2_1_39_1","unstructured":"J. Nielsen. 2005b. How to Conduct a Heuristic Valuation. Retrieved May 8 2015 from http:\/\/www.useit.com\/ papers\/heuristic\/heuristic_evaluation.html.  J. Nielsen. 2005b. How to Conduct a Heuristic Valuation. Retrieved May 8 2015 from http:\/\/www.useit.com\/ papers\/heuristic\/heuristic_evaluation.html."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2004.1297314"},{"key":"e_1_2_1_42_1","unstructured":"J. F. Pane. 2002. A programming system for children that is designed for usability. Ph.D. Dissertation. Carnegie Mellon University Pittsburgh Pennsylvania.   J. F. Pane. 2002. A programming system for children that is designed for usability. Ph.D. Dissertation. Carnegie Mellon University Pittsburgh Pennsylvania."},{"volume-title":"PA","year":"1996","author":"Pane J. F.","key":"e_1_2_1_43_1"},{"volume-title":"Mindstorms: Children, Computers, and Powerful Ideas","year":"1980","author":"Papert S.","key":"e_1_2_1_44_1"},{"key":"e_1_2_1_45_1","unstructured":"R. E. Pattis. 1981. Karel the Robot: A Gentle Introduction to the Art of Programming (1st ed.). John Wiley & Sons New York NY.   R. E. Pattis. 1981. Karel the Robot: A Gentle Introduction to the Art of Programming (1st ed.). John Wiley & Sons New York NY."},{"volume-title":"Retrieved","year":"2012","author":"Foundation Python Software","key":"e_1_2_1_46_1"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/5600.5702"},{"volume-title":"Proceedings of the 8th Australian Conference on Computing Education. 165--173","author":"Robins A.","key":"e_1_2_1_48_1"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/611892.611968"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1269900.1268834"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1151588.1151593"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327590ijhc0903_2"},{"key":"e_1_2_1_53_1","unstructured":"S. Shing\u014d. 1986. Zero Quality Control: Source Inspection and the Poka-Yoke System. Productivity Press New York NY 305 pages.  S. Shing\u014d. 1986. Zero Quality Control: Source Inspection and the Poka-Yoke System. Productivity Press New York NY 305 pages."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124706.1121427"},{"key":"e_1_2_1_55_1","unstructured":"J. M. Slack. 1990. Turbo Pascal with Turtle Graphics. West Publishing Co. St. Paul MN.   J. M. Slack. 1990. Turbo Pascal with Turtle Graphics. West Publishing Co. St. Paul MN."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/611892.612007"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1140124.1140135"},{"key":"e_1_2_1_58_1","first-page":"3","article-title":"Epistemological pluralism and the revaluation of the concrete","volume":"11","author":"Turkle S.","year":"1992","journal-title":"J. Math. Behav."},{"key":"e_1_2_1_59_1","unstructured":"C. Ware. 2008. Visual Thinking for Design. Morgan Kaufmann Burlington MA.   C. Ware. 2008. Visual Thinking for Design. Morgan Kaufmann Burlington MA."},{"key":"e_1_2_1_60_1","unstructured":"G. M. Weinberg. 1998. The Psychology of Computer Programming (2nd ed.). Dorset House Publishing New York NY.   G. M. Weinberg. 1998. The Psychology of Computer Programming (2nd ed.). Dorset House Publishing New York NY."}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2872521","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2872521","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:00:38Z","timestamp":1750230038000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2872521"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,6,8]]},"references-count":60,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2016,6,27]]}},"alternative-id":["10.1145\/2872521"],"URL":"https:\/\/doi.org\/10.1145\/2872521","relation":{},"ISSN":["1946-6226"],"issn-type":[{"type":"electronic","value":"1946-6226"}],"subject":[],"published":{"date-parts":[[2016,6,8]]},"assertion":[{"value":"2015-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-06-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}