{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T15:14:50Z","timestamp":1779117290529,"version":"3.51.4"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2002,6,24]],"date-time":"2002-06-24T00:00:00Z","timestamp":1024876800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGCSE Bull."],"published-print":{"date-parts":[[2003,6]]},"abstract":"<jats:p>Visualization technology can be used to graphically illustrate various concepts in computer science. We argue that such technology, no matter how well it is designed, is of little educational value unless it engages learners in an active learning activity. Drawing on a review of experimental studies of visualization effectiveness, we motivate this position against the backdrop of current attitudes and best practices with respect to visualization use. We suggest a new taxonomy of learner engagement with visualization technology. Grounded in Bloom's well-recognized taxonomy of understanding, we suggest metrics for assessing the learning outcomes to which such engagement may lead. Based on these taxonomies of engagement and effectiveness metrics, we present a framework for experimental studies of visualization effectiveness. Interested computer science educators are invited to collaborate with us by carrying out studies within this framework.<\/jats:p>","DOI":"10.1145\/782941.782998","type":"journal-article","created":{"date-parts":[[2004,7,21]],"date-time":"2004-07-21T22:43:36Z","timestamp":1090449816000},"page":"131-152","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":308,"title":["Exploring the role of visualization and engagement in computer science education"],"prefix":"10.1145","volume":"35","author":[{"given":"Thomas L.","family":"Naps","sequence":"first","affiliation":[{"name":"U Wisconsin Oshkosh"}]},{"given":"Guido","family":"R\u00f6\u00dfling","sequence":"additional","affiliation":[{"name":"Darmstadt U Techn., Germany"}]},{"given":"Vicki","family":"Almstrum","sequence":"additional","affiliation":[{"name":"U Texas Austin"}]},{"given":"Wanda","family":"Dann","sequence":"additional","affiliation":[{"name":"Ithaca College"}]},{"given":"Rudolf","family":"Fleischer","sequence":"additional","affiliation":[{"name":"Hong Kong U Sc. &amp; Techn."}]},{"given":"Chris","family":"Hundhausen","sequence":"additional","affiliation":[{"name":"U of Hawaii"}]},{"given":"Ari","family":"Korhonen","sequence":"additional","affiliation":[{"name":"Helsinki U Techn., Finland"}]},{"given":"Lauri","family":"Malmi","sequence":"additional","affiliation":[{"name":"Helsinki U Techn., Finland"}]},{"given":"Myles","family":"McNally","sequence":"additional","affiliation":[{"name":"Alma College"}]},{"given":"Susan","family":"Rodger","sequence":"additional","affiliation":[{"name":"Duke University"}]},{"given":"J. \u00c1ngel","family":"Vel\u00e1zquez-Iturbide","sequence":"additional","affiliation":[{"name":"U Rey Juan Carlos, Spain"}]}],"member":"320","published-online":{"date-parts":[[2002,6,24]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"First International Program Visualization Workshop","author":"Anderson J. M.","year":"2001","unstructured":"Anderson , J. M. , and Naps , T. L . A Context for the Assessment of Algorithm Visualization System as Pedagogical Tools . First International Program Visualization Workshop , Porvoo, Finland. University of Joensuu Press ( July 2001 ), 121--130. Anderson, J. M., and Naps, T. L. A Context for the Assessment of Algorithm Visualization System as Pedagogical Tools. First International Program Visualization Workshop, Porvoo, Finland. University of Joensuu Press (July 2001), 121--130."},{"key":"e_1_2_1_2_1","first-page":"369","volume-title":"Eds. MIT Press","author":"Baecker R.","year":"1998","unstructured":"Baecker , R. Sorting Out Sorting: A Case Study of Software Visualization for Teaching Computer Science. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price , Eds. MIT Press , 1998 , ch. 24, pp. 369 -- 381 . Baecker, R. Sorting Out Sorting: A Case Study of Software Visualization for Teaching Computer Science. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price, Eds. MIT Press, 1998, ch. 24, pp. 369--381."},{"key":"e_1_2_1_3_1","first-page":"382","volume-title":"Eds. MIT Press","author":"Bazik J.","year":"1998","unstructured":"Bazik , J. , Tamassia , R. , Reiss , S. P. , and van Dam , A. Software Visualization in Teaching at Brown University. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price , Eds. MIT Press , 1998 , ch. 25, pp. 382 -- 398 . Bazik, J., Tamassia, R., Reiss, S. P., and van Dam, A. Software Visualization in Teaching at Brown University. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price, Eds. MIT Press, 1998, ch. 25, pp. 382--398."},{"key":"e_1_2_1_4_1","first-page":"778","author":"Biermann H.","year":"1999","unstructured":"Biermann , H. , and Cole , R. Comic Strips for Algorithm Visualization. Tech. rep., NYU 1999 -- 778 , New York University, Feb. 1999. Biermann, H., and Cole, R. Comic Strips for Algorithm Visualization. Tech. rep., NYU 1999--778, New York University, Feb. 1999.","journal-title":"Visualization. Tech. rep., NYU"},{"key":"e_1_2_1_5_1","volume-title":"Taxonomy of Educational Objectives","author":"Bloom B. S.","year":"1956","unstructured":"Bloom , B. S. , and Krathwohl , D. R . Taxonomy of Educational Objectives ; the Classification of Educational Goals, Handbook I : Cognitive Domain. AddisonWesley , 1956 . Bloom, B. S., and Krathwohl, D. R. Taxonomy of Educational Objectives; the Classification of Educational Goals, Handbook I: Cognitive Domain. AddisonWesley, 1956."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/236452.236526"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/330908.331843"},{"key":"e_1_2_1_8_1","unstructured":"Brown M. H. Algorithm Animation. MIT Press Cambridge Massachussets 1988.  Brown M. H. Algorithm Animation. MIT Press Cambridge Massachussets 1988."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(97)00090-1"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/800031.808596"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(99)00023-8"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0360-1315(96)00006-1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/218683.218689"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/377435.377507"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/647382.760295"},{"key":"e_1_2_1_16_1","first-page":"5","volume":"23","author":"Felder R. M.","year":"1993","unstructured":"Felder , R. M. Reaching the second tier. Journal of College Science Teaching 23 , 5 ( 1993 ), 286--290. Felder, R. M. Reaching the second tier. Journal of College Science Teaching 23, 5 (1993), 286--290.","journal-title":"Journal of College Science Teaching"},{"key":"e_1_2_1_17_1","first-page":"4","volume":"6","author":"Felder R. M.","year":"1996","unstructured":"Felder , R. M. Matters of style. ASEE Prism 6 , 4 ( 1996 ), 18--23. Felder, R. M. Matters of style. ASEE Prism 6, 4 (1996), 18--23.","journal-title":"ASEE Prism"},{"key":"e_1_2_1_18_1","first-page":"145","volume-title":"Eds. MIT Press","author":"Gloor P. A.","year":"1998","unstructured":"Gloor , P. A. User Interface Issues for Algorithm Animation. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price , Eds. MIT Press , 1998 , ch. 11, pp. 145 -- 152 . Gloor, P. A. User Interface Issues for Algorithm Animation. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price, Eds. MIT Press, 1998, ch. 11, pp. 145--152."},{"key":"e_1_2_1_19_1","volume-title":"Personal communication","author":"Grissom S.","year":"2002","unstructured":"Grissom , S. Personal communication , 2002 . Grissom, S. Personal communication, 2002."},{"key":"e_1_2_1_21_1","volume-title":"Animation of User Algorithms on the Web. IEEE Symposium on Visual Languages","author":"Haajanen J.","year":"1997","unstructured":"Haajanen , J. , Pesonius , M. , Sutinen , E. , Tarhio , J. , Ter\u00e4svirta , T. , and Vanninen , P . Animation of User Algorithms on the Web. IEEE Symposium on Visual Languages ( 1997 ), 360--367. Haajanen, J., Pesonius, M., Sutinen, E., Tarhio, J., Ter\u00e4svirta, T., and Vanninen, P. Animation of User Algorithms on the Web. IEEE Symposium on Visual Languages (1997), 360--367."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA 1999","author":"Hansen S.","year":"1999","unstructured":"Hansen , S. , Schrimpsher , D. , and Narayanan , N. H ., From Algorithm Animations to Animation-Embedded Hypermedia Visualizations . Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA 1999 ), Seattle, Washington ( 1999 ), 1032--1037. Hansen, S., Schrimpsher, D., and Narayanan, N. H., From Algorithm Animations to Animation-Embedded Hypermedia Visualizations. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA 1999), Seattle, Washington (1999), 1032--1037."},{"key":"e_1_2_1_23_1","first-page":"1","volume":"2","author":"Hansen S. R.","year":"2000","unstructured":"Hansen , S. R. , Narayanan , N. H. , and Schrimpsher , D. Helping Learners Visualize and Comprehend Algorithms. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning 2 , 1 ( 2000 ). Hansen, S. R., Narayanan, N. H., and Schrimpsher, D. Helping Learners Visualize and Comprehend Algorithms. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning 2, 1 (2000).","journal-title":"Helping Learners Visualize and Comprehend Algorithms. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(02)00044-1"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/832281.834570"},{"key":"e_1_2_1_27_1","volume-title":"Low-Fidelity Algorithm Visualization. Journal of Visual Languages and Computing","author":"Hundhausen C. D.","year":"2002","unstructured":"Hundhausen , C. D. , and Douglas , S. A . Low-Fidelity Algorithm Visualization. Journal of Visual Languages and Computing ( 2002 ), (in print). Hundhausen, C. D., and Douglas, S. A. Low-Fidelity Algorithm Visualization. Journal of Visual Languages and Computing (2002), (in print)."},{"key":"e_1_2_1_28_1","volume-title":"A Meta-Study of Algorithm Visualization Effectiveness. Journal of Visual Languages and Computing","author":"Hundhausen C. D.","year":"2002","unstructured":"Hundhausen , C. D. , Douglas , S. A. , and Stasko , J. T . A Meta-Study of Algorithm Visualization Effectiveness. Journal of Visual Languages and Computing ( 2002 ), (in print). Hundhausen, C. D., Douglas, S. A., and Stasko, J. T. A Meta-Study of Algorithm Visualization Effectiveness. Journal of Visual Languages and Computing (2002), (in print)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/330908.331800"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/330908.331889"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(97)00015-8"},{"key":"e_1_2_1_32_1","volume-title":"Nov. 3","author":"Katz B.","year":"1998","unstructured":"Katz , B. , and Almstrum , V . Collaborative Project Plans, version 1.3. WWW: http:\/\/www.cs.utexas.edu\/users\/csed\/CPP\/ , Nov. 3 , 1998 . Katz, B., and Almstrum, V. Collaborative Project Plans, version 1.3. WWW: http:\/\/www.cs.utexas.edu\/users\/csed\/CPP\/, Nov. 3, 1998."},{"key":"e_1_2_1_33_1","volume-title":"Keirsey Temperament and Character Web Site. WWW: http:\/\/www.keirsey.com","author":"Keirsey D. M.","year":"2002","unstructured":"Keirsey , D. M. Keirsey Temperament and Character Web Site. WWW: http:\/\/www.keirsey.com , 2002 . Keirsey, D. M. Keirsey Temperament and Character Web Site. WWW: http:\/\/www.keirsey.com, 2002."},{"key":"e_1_2_1_34_1","volume-title":"S. Designing Effective Algorithm Visualizations. First International Program Visualization Workshop","author":"Khuri","year":"2001","unstructured":"Khuri , S. Designing Effective Algorithm Visualizations. First International Program Visualization Workshop , Porvoo, Finland. University of Joensuu Press ( Feb. 2001 ), 1--12. Khuri, S. Designing Effective Algorithm Visualizations. First International Program Visualization Workshop, Porvoo, Finland. University of Joensuu Press (Feb. 2001), 1--12."},{"key":"e_1_2_1_35_1","volume-title":"Special Issue on Visualization of Software (Apr.","author":"Khuri S.","year":"2001","unstructured":"Khuri , S. A User-Centered Approach for Designing Algorithm Visualizations. Informatik \/ Informatique , Special Issue on Visualization of Software (Apr. 2001 ), 12--16. Khuri, S. A User-Centered Approach for Designing Algorithm Visualizations. Informatik \/ Informatique, Special Issue on Visualization of Software (Apr. 2001), 12--16."},{"key":"e_1_2_1_36_1","volume-title":"Experiential Learning","author":"Kolb D.","year":"1984","unstructured":"Kolb , D. Experiential Learning . Prentice-Hall , New Jersey , 1984 . Kolb, D. Experiential Learning. Prentice-Hall, New Jersey, 1984."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/343048.343157"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1556262.1556278"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/647382.724663"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.83.4.484"},{"key":"e_1_2_1_42_1","volume-title":"Second International Program Visualization Workshop, \u00c4rhus, Denmark (June","author":"Naharro-Berrocal F.","year":"2002","unstructured":"Naharro-Berrocal , F. , Pareja-Flores , C. , Urquiza-Fuentes , J. , Vel\u00e1zquez-Iturbide , J. A. , and Gort\u00e1zar-Bellas , F . Redesigning the Animation Capabilities of a Functional Programming Environment under an Educational Framework . Second International Program Visualization Workshop, \u00c4rhus, Denmark (June 2002 ), (in print). Naharro-Berrocal, F., Pareja-Flores, C., Urquiza-Fuentes, J., Vel\u00e1zquez-Iturbide, J. A., and Gort\u00e1zar-Bellas, F. Redesigning the Animation Capabilities of a Functional Programming Environment under an Educational Framework. Second International Program Visualization Workshop, \u00c4rhus, Denmark (June 2002), (in print)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/1253530.1254554"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/330908.331829"},{"key":"e_1_2_1_45_1","volume-title":"Special Issue on Visualization of Software (Apr.","author":"Naps T. L.","year":"2001","unstructured":"Naps , T. L. Incorporating Algorithm Visualization into Educational Theory: A Challenge for the Future. Informatik \/ Informatique , Special Issue on Visualization of Software (Apr. 2001 ), 17--21. Naps, T. L. Incorporating Algorithm Visualization into Educational Theory: A Challenge for the Future. Informatik \/ Informatique, Special Issue on Visualization of Software (Apr. 2001), 17--21."},{"key":"e_1_2_1_46_1","volume-title":"International Journal of Psychology in the Orient","author":"Niemivirta M.","year":"2002","unstructured":"Niemivirta , M. Motivation and performance in context - the influence of goal orientation and instructional setting on situational appraisals and task performance . International Journal of Psychology in the Orient ( 2002 ), (in print). Niemivirta, M. Motivation and performance in context - the influence of goal orientation and instructional setting on situational appraisals and task performance. International Journal of Psychology in the Orient (2002), (in print)."},{"key":"e_1_2_1_47_1","volume-title":"Mindstorms: Children, Computers, and Powerful Ideas","author":"Papert S.","year":"1980","unstructured":"Papert , S. Mindstorms: Children, Computers, and Powerful Ideas . Basic Books , New York , 1980 . Papert, S. Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, New York, 1980."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/273133.274310"},{"key":"e_1_2_1_49_1","first-page":"3","volume-title":"Eds. MIT Press","author":"Price B.","year":"1998","unstructured":"Price , B. , Baecker , R. , and Small , I . An Introduction to Software Visualization. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price , Eds. MIT Press , 1998 , ch. 1, pp. 3 -- 27 . Price, B., Baecker, R., and Small, I. An Introduction to Software Visualization. In Software Visualization, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price, Eds. MIT Press, 1998, ch. 1, pp. 3--27."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1006\/jvlc.1993.1015"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/237466.237533"},{"key":"e_1_2_1_52_1","volume-title":"Personal communication","author":"Ross R. J.","year":"2002","unstructured":"Ross , R. J. Personal communication , 2002 . Ross, R. J. Personal communication, 2002."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1006\/jvlc.2002.0239"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.58216"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/268084.268091"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169078"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/305786.305891"}],"container-title":["ACM SIGCSE Bulletin"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/782941.782998","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/782941.782998","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:43:58Z","timestamp":1750268638000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/782941.782998"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2002,6,24]]},"references-count":54,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2003,6]]}},"alternative-id":["10.1145\/782941.782998"],"URL":"https:\/\/doi.org\/10.1145\/782941.782998","relation":{"is-identical-to":[{"id-type":"doi","id":"10.1145\/960568.782998","asserted-by":"subject"}]},"ISSN":["0097-8418"],"issn-type":[{"value":"0097-8418","type":"print"}],"subject":[],"published":{"date-parts":[[2002,6,24]]},"assertion":[{"value":"2002-06-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}