{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T05:07:07Z","timestamp":1771045627008,"version":"3.50.1"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2010,1,18]],"date-time":"2010-01-18T00:00:00Z","timestamp":1263772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGCSE Bull."],"published-print":{"date-parts":[[2010,1,18]]},"abstract":"<jats:p>In New Zealand and Australia, the BRACElet project has been investigating students' acquisition of programming skills in introductory programming courses. The project has explored students' skills in basic syntax, tracing code, understanding code, and writing code, seeking to establish the relationships between these skills. This ITiCSE working group report presents the most recent step in the BRACElet project, which includes replication of earlier analysis using a far broader pool of naturally occurring data, refinement of the SOLO taxonomy in code-explaining questions, extension of the taxonomy to code-writing questions, extension of some earlier studies on students' 'doodling' while answering exam questions, and exploration of a further theoretical basis for work that until now has been primarily empirical.<\/jats:p>","DOI":"10.1145\/1709424.1709460","type":"journal-article","created":{"date-parts":[[2010,1,26]],"date-time":"2010-01-26T14:01:38Z","timestamp":1264514498000},"page":"156-173","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":69,"title":["Naturally occurring data as research instrument"],"prefix":"10.1145","volume":"41","author":[{"given":"Raymond","family":"Lister","sequence":"first","affiliation":[{"name":"University of British Columbia, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tony","family":"Clear","sequence":"additional","affiliation":[{"name":"AUT University, New Zealand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"family":"Simon","sequence":"additional","affiliation":[{"name":"University of Newcastle, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dennis J.","family":"Bouvier","sequence":"additional","affiliation":[{"name":"Southern Illinois University, Edwardsville"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul","family":"Carter","sequence":"additional","affiliation":[{"name":"University of British Columbia, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anna","family":"Eckerdal","sequence":"additional","affiliation":[{"name":"Uppsala University, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jana","family":"Jackov\u00e1","sequence":"additional","affiliation":[{"name":"University of Zilina, Slovakia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mike","family":"Lopez","sequence":"additional","affiliation":[{"name":"Manukau Institute of Technology, New Zealand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"McCartney","sequence":"additional","affiliation":[{"name":"University of Connecticut"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Phil","family":"Robbins","sequence":"additional","affiliation":[{"name":"AUT University, New Zealand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Otto","family":"Sepp\u00e4l\u00e4","sequence":"additional","affiliation":[{"name":"Helsinki University of Technology, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Errol","family":"Thompson","sequence":"additional","affiliation":[{"name":"England"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2010,1,18]]},"reference":[{"issue":"2","key":"e_1_2_1_1_1","first-page":"115","article-title":"An alternative reconceptualization of procedural and conceptual knowledge","volume":"38","author":"Baroody A.","year":"2007","unstructured":"Baroody , A. , Feil , Y. , and Johnson , A. ( 2007 ). An alternative reconceptualization of procedural and conceptual knowledge . Journal for Research in Mathematics Education , 38 2 , 115 -- 131 . Baroody, A., Feil, Y., and Johnson, A. (2007). An alternative reconceptualization of procedural and conceptual knowledge. Journal for Research in Mathematics Education, 38:2, 115--131.","journal-title":"Journal for Research in Mathematics Education"},{"key":"e_1_2_1_2_1","volume-title":"Teaching for quality learning at university","author":"Biggs J. B.","year":"1999","unstructured":"Biggs , J. B. ( 1999 ). Teaching for quality learning at university . Buckingham : Open University Press . Biggs, J. B. (1999). Teaching for quality learning at university. Buckingham: Open University Press."},{"key":"e_1_2_1_3_1","volume-title":"Evaluating the quality of learning: the SOLO taxonomy (Structure of the Observed Learning Outcome)","author":"Biggs J. B.","year":"1982","unstructured":"Biggs , J. B. and Collis , K. F . ( 1982 ). Evaluating the quality of learning: the SOLO taxonomy (Structure of the Observed Learning Outcome) . New York : Academic Press . Biggs, J. B. and Collis, K. F. (1982). Evaluating the quality of learning: the SOLO taxonomy (Structure of the Observed Learning Outcome). New York: Academic Press."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124706.1121470"},{"key":"e_1_2_1_5_1","volume-title":"Tenth Australasian Computing Education Conference (ACE","author":"Clear T.","year":"2008","unstructured":"Clear , T. , Edwards , J. , Lister , R. , Simon , B. , Thompson , E. and Whalley , J . ( 2008 ). The teaching of novice computer programmers: bringing the scholarly-research approach to Australia . Tenth Australasian Computing Education Conference (ACE 2008), Wollongong, Australia, 63--68. Clear, T., Edwards, J., Lister, R., Simon, B., Thompson, E. and Whalley, J. (2008). The teaching of novice computer programmers: bringing the scholarly-research approach to Australia. Tenth Australasian Computing Education Conference (ACE 2008), Wollongong, Australia, 63--68."},{"key":"e_1_2_1_6_1","volume-title":"21st Annual NACCQ Conference, NACCQ","author":"Clear T.","year":"2008","unstructured":"Clear , T. , Whalley , J. , Lister , R. , Carbone , A. , Hu , M. , Sheard , J. , Simon , B. and Thompson , E . ( 2008 ). Reliably classifying novice programmer exam results using the SOLO taxonomy . 21st Annual NACCQ Conference, NACCQ , Auckland, New Zealand, 23--30. Clear, T., Whalley, J., Lister, R., Carbone, A., Hu, M., Sheard, J., Simon, B. and Thompson, E. (2008). Reliably classifying novice programmer exam results using the SOLO taxonomy. 21st Annual NACCQ Conference, NACCQ, Auckland, New Zealand, 23--30."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-007-9061-7"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0732-3123(96)90015-2"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404532"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1089786.1089797"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1003640204118"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF03217454"},{"key":"e_1_2_1_13_1","volume-title":"Dart, B. & Boulton-Lewis","author":"Hattie J.","year":"1998","unstructured":"Hattie , J. and Purdie , N . ( 1998 ). The SOLO model: addressing fundamental measurement issues . In Dart, B. & Boulton-Lewis , G. M. (Eds.), Teaching and learning in higher education, 145--176. Camberwell, Vic : Australian Council of Educational Research . Hattie, J. and Purdie, N. (1998). The SOLO model: addressing fundamental measurement issues. In Dart, B. & Boulton-Lewis, G. M. (Eds.), Teaching and learning in higher education, 145--176. Camberwell, Vic: Australian Council of Educational Research."},{"key":"e_1_2_1_14_1","volume-title":"Conceptual and procedural knowledge: the case of mathematics, 1--27","author":"Hiebert J.","year":"1986","unstructured":"Hiebert , J. and Lefevre , P . ( 1986 ). Conceptual and procedural knowledge in mathematics: an introductory analysis . In Hiebert, J., ed., Conceptual and procedural knowledge: the case of mathematics, 1--27 . Erlbaum , Hillsdale, NJ . Hiebert, J. and Lefevre, P. (1986). Conceptual and procedural knowledge in mathematics: an introductory analysis. In Hiebert, J., ed., Conceptual and procedural knowledge: the case of mathematics, 1--27. Erlbaum, Hillsdale, NJ."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1041624.1041673"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1562877.1562930"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1140123.1140157"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404531"},{"key":"e_1_2_1_19_1","volume-title":"Fourth Finnish \/ Baltic Sea Conference on Computer Science Education (Koli Calling","author":"McCartney R.","year":"2004","unstructured":"McCartney , R. , Mostrom , J.E. , Sanders , K. , and Sepp\u00e4l\u00e4 , O . ( 2004 ). Questions, annotations, and institutions: observations from a study of novice programmers . Fourth Finnish \/ Baltic Sea Conference on Computer Science Education (Koli Calling 2004), 11--19. McCartney, R., Mostrom, J.E., Sanders, K., and Sepp\u00e4l\u00e4, O. (2004). Questions, annotations, and institutions: observations from a study of novice programmers. Fourth Finnish \/ Baltic Sea Conference on Computer Science Education (Koli Calling 2004), 11--19."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008819912213"},{"key":"e_1_2_1_21_1","volume-title":"Eighth Australasian Computing Education Conference (ACE2006)","author":"Parsons D.","year":"2006","unstructured":"Parsons , D. and Haden , P . ( 2006 ). Parson's programming puzzles: a fun and effective learning tool for first programming courses . Eighth Australasian Computing Education Conference (ACE2006) , Hobart, Australia, 157--163. Parsons, D. and Haden, P. (2006). Parson's programming puzzles: a fun and effective learning tool for first programming courses. Eighth Australasian Computing Education Conference (ACE2006), Hobart, Australia, 157--163."},{"key":"e_1_2_1_22_1","volume-title":"Mathematical Cognition 227--259","author":"Pegg J.","year":"2002","unstructured":"Pegg , J. ( 2002 ). Assessment in mathematics: A developmental approach . In J. M. Royer (Ed.), Mathematical Cognition 227--259 . Information Age Publishing : Greenwich, CT, USA . Pegg, J. (2002). Assessment in mathematics: A developmental approach. In J. M. Royer (Ed.), Mathematical Cognition 227--259. Information Age Publishing: Greenwich, CT, USA."},{"issue":"6","key":"e_1_2_1_23_1","first-page":"468","article-title":"The fundamental cycle of concept construction underlying various theoretical frameworks","volume":"37","author":"Pegg J.","year":"2005","unstructured":"Pegg , J. and Tall , D. ( 2005 ). The fundamental cycle of concept construction underlying various theoretical frameworks . ZDM 37 6 , 468 -- 474 . Pegg, J. and Tall, D. (2005). The fundamental cycle of concept construction underlying various theoretical frameworks. ZDM 37:6, 468--474.","journal-title":"ZDM"},{"key":"e_1_2_1_24_1","volume-title":"Studying the Novice Programmer","author":"Perkins D.","year":"1989","unstructured":"Perkins , D. and Martin , F . ( 1989 ). Fragile knowledge and neglected strategies in novice programmers . In Soloway, E. and Spohrer, J, Eds., Studying the Novice Programmer . Lawrence Erlbaum Associates , Hillsdale, NJ, USA , 213--229. Perkins, D. and Martin, F. (1989). Fragile knowledge and neglected strategies in novice programmers. In Soloway, E. and Spohrer, J, Eds., Studying the Novice Programmer. Lawrence Erlbaum Associates, Hillsdale, NJ, USA, 213--229."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/DASC.2007.15"},{"key":"e_1_2_1_26_1","volume-title":"Probabilistic models for some intelligence and attainment tests (Copenhagen","author":"Rasch G.","year":"1980","unstructured":"Rasch , G. (1960\/1980). Probabilistic models for some intelligence and attainment tests (Copenhagen , Danish Institute for Educational Research) , expanded edition ( 1980 ) with foreword and afterword by B.D. Wright. Chicago : The University of Chicago Press . Rasch, G. (1960\/1980). Probabilistic models for some intelligence and attainment tests (Copenhagen, Danish Institute for Educational Research), expanded edition (1980) with foreword and afterword by B.D. Wright. Chicago: The University of Chicago Press."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1076\/csed.13.2.137.14200"},{"key":"e_1_2_1_28_1","volume-title":"Twelfth Conference for the Psychology of Mathematics Education 2, 560--567","author":"Sfard A.","year":"1988","unstructured":"Sfard , A. ( 1988 ). Operational vs. structural method of teaching mathematics -- case study . Twelfth Conference for the Psychology of Mathematics Education 2, 560--567 . Vespr\u00e8m, Hungary : Ferenc Genzwein, OOK. Sfard, A. (1988). Operational vs. structural method of teaching mathematics -- case study. Twelfth Conference for the Psychology of Mathematics Education 2, 560--567. Vespr\u00e8m, Hungary: Ferenc Genzwein, OOK."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00302715"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1597849.1384328"},{"key":"e_1_2_1_31_1","volume-title":"Ninth International Conference on Computing Education Research (Koli Calling","author":"Simon","year":"2009","unstructured":"Simon ( 2009 ). Code-explaining exam questions: a cautionary note . Ninth International Conference on Computing Education Research (Koli Calling 2009). Simon (2009). Code-explaining exam questions: a cautionary note. Ninth International Conference on Computing Education Research (Koli Calling 2009)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/6592.6594"},{"key":"e_1_2_1_33_1","first-page":"401","article-title":"Reconceptualizing procedural knowledge","volume":"36","author":"Star J. R.","year":"2005","unstructured":"Star , J. R. ( 2005 ). Reconceptualizing procedural knowledge . Journal for Research in Mathematics Education 36 , 401 -- 411 . Star, J. R. (2005). Reconceptualizing procedural knowledge. Journal for Research in Mathematics Education 36, 401--411.","journal-title":"Journal for Research in Mathematics Education"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF03217474"},{"key":"e_1_2_1_35_1","volume-title":"21st Annual NACCQ Conference, NACCQ","author":"Thompson E.","year":"2004","unstructured":"Thompson , E. ( 2004 ). Does the sum of the parts equal the whole? 21st Annual NACCQ Conference, NACCQ , Auckland, New Zealand , 2008, 440--445. Thompson, E. (2004). Does the sum of the parts equal the whole? 21st Annual NACCQ Conference, NACCQ, Auckland, New Zealand, 2008, 440--445."},{"key":"e_1_2_1_36_1","volume-title":"Ninth Australasian Computing Education Conference (ACE2007)","author":"Thompson E.","year":"2007","unstructured":"Thompson , E. ( 2007 ). Holistic assessment criteria -- Applying SOLO to programming projects . Ninth Australasian Computing Education Conference (ACE2007) , Ballarat, Australia, 155--162. Thompson, E. (2007). Holistic assessment criteria -- Applying SOLO to programming projects. Ninth Australasian Computing Education Conference (ACE2007), Ballarat, Australia, 155--162."},{"key":"e_1_2_1_37_1","volume-title":"How do they understand? Practitioner perceptions of an object-oriented program. Dissertation","author":"Thompson E.","year":"2009","unstructured":"Thompson , E. ( 2009 ). How do they understand? Practitioner perceptions of an object-oriented program. Dissertation , Massey University , Palmerston North . Thompson, E. (2009). How do they understand? Practitioner perceptions of an object-oriented program. Dissertation, Massey University, Palmerston North."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1584322.1584336"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-7373(85)80041-9"},{"key":"e_1_2_1_40_1","volume-title":"Eighth Australasian Computing Education Conference (ACE2006)","author":"Whalley J. L.","year":"2006","unstructured":"Whalley , J. L. , Lister , R. , Thompson , E. , Clear , T. , Robbins , P. , Kumar , P. K. A. , and Prasad , C . ( 2006 ). An Australasian study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies . Eighth Australasian Computing Education Conference (ACE2006) , Hobart, Australia, 243--252. Whalley, J. L., Lister, R., Thompson, E., Clear, T., Robbins, P., Kumar, P. K. A., and Prasad, C. (2006). An Australasian study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies. Eighth Australasian Computing Education Conference (ACE2006), Hobart, Australia, 243--252."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/1862712.1862717"},{"key":"e_1_2_1_42_1","volume-title":"Report on the Fourth BRACElet Workshop. Bulletin of Applied Computing and IT. Retrieved","author":"Whalley J.","year":"2007","unstructured":"Whalley , J. and Robbins , P . ( 2007 ). Report on the Fourth BRACElet Workshop. Bulletin of Applied Computing and IT. Retrieved June 7, 2007 from http:\/\/www.naccq.co.nz\/bacit\/0501\/2007Whalley_BRACELET_Workshop.htm. Whalley, J. and Robbins, P. (2007). Report on the Fourth BRACElet Workshop. Bulletin of Applied Computing and IT. Retrieved June 7, 2007 from http:\/\/www.naccq.co.nz\/bacit\/0501\/2007Whalley_BRACELET_Workshop.htm."},{"issue":"3","key":"e_1_2_1_43_1","first-page":"370","article-title":"Reasonable meansquare fit values","volume":"8","author":"Wright B.D","year":"1994","unstructured":"Wright , B.D , and Linacre , J.M. ( 1994 ) Reasonable meansquare fit values . Rasch Measurement Transactions 8 3 , 370 . Wright, B.D, and Linacre, J.M. (1994) Reasonable meansquare fit values. Rasch Measurement Transactions 8:3, 370.","journal-title":"Rasch Measurement Transactions"}],"container-title":["ACM SIGCSE Bulletin"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1709424.1709460","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1709424.1709460","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:22:10Z","timestamp":1750278130000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1709424.1709460"}},"subtitle":["analyzing examination responses to study the novice programmer"],"short-title":[],"issued":{"date-parts":[[2010,1,18]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2010,1,18]]}},"alternative-id":["10.1145\/1709424.1709460"],"URL":"https:\/\/doi.org\/10.1145\/1709424.1709460","relation":{},"ISSN":["0097-8418"],"issn-type":[{"value":"0097-8418","type":"print"}],"subject":[],"published":{"date-parts":[[2010,1,18]]},"assertion":[{"value":"2010-01-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}