{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,3]],"date-time":"2026-05-03T03:14:26Z","timestamp":1777778066517,"version":"3.51.4"},"reference-count":29,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2013,6,11]],"date-time":"2013-06-11T00:00:00Z","timestamp":1370908800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Information Visualization"],"published-print":{"date-parts":[[2014,10]]},"abstract":"<jats:p>Evaluation of software can take many forms ranging from algorithm correctness and performance to evaluations that focus on the value to the end user. This article presents a discussion of the development of an evaluation methodology for visual analytics environments. The Visual Analytics Science and Technology Challenge was created as a community evaluation resource. This resource is available to researchers and developers of visual analytics environments and allows them to test out their designs and visualization and compare the results with the solution and the entries prepared by others. Sharing results allows the community to learn from each other and to hopefully advance more quickly. In this article, we discuss the original challenge and its evolution during the 7 years since its inception. While the Visual Analytics Science and Technology Challenge is the focus of this article, there are lessons for many involved in setting up a community evaluation program, including the need to understand the purpose of the evaluation, decide upon the right metrics to use, and the appropriate implementation of those metrics including datasets and evaluators. For ongoing evaluations, it is also necessary to track the evolution and to ensure that the evaluation methodologies are keeping pace with the science being evaluated. The discussions on the Visual Analytics Science and Technology Challenge on these topics should be pertinent to many interested in community evaluations.<\/jats:p>","DOI":"10.1177\/1473871613490290","type":"journal-article","created":{"date-parts":[[2013,6,12]],"date-time":"2013-06-12T00:16:10Z","timestamp":1370996170000},"page":"326-335","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":19,"title":["Evaluation of visual analytics environments: The road to the Visual Analytics Science and Technology challenge evaluation methodology"],"prefix":"10.1177","volume":"13","author":[{"given":"Jean","family":"Scholtz","sequence":"first","affiliation":[{"name":"Pacific Northwest National Laboratory, Richland, WA, USA"}]},{"given":"Catherine","family":"Plaisant","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD, USA"}]},{"given":"Mark","family":"Whiting","sequence":"additional","affiliation":[{"name":"Pacific Northwest National Laboratory, Richland, WA, USA"}]},{"given":"Georges","family":"Grinstein","sequence":"additional","affiliation":[{"name":"University of Massachusetts Lowell, Lowell, MA, USA"}]}],"member":"179","published-online":{"date-parts":[[2013,6,11]]},"reference":[{"key":"bibr1-1473871613490290","volume-title":"Illuminating the path: the research and development agenda for visual analytics","author":"Thomas J","year":"2005"},{"key":"bibr2-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2011.279"},{"key":"bibr3-1473871613490290","first-page":"19","volume-title":"Information visualization: human-centered issues and perspectives","author":"Carpendale S.","year":"2007"},{"key":"bibr4-1473871613490290","first-page":"109","volume-title":"Proceedings of the working conference on advanced visual interfaces","author":"Plaisant C."},{"issue":"3","key":"bibr5-1473871613490290","first-page":"409","volume-title":"Comput Linguist","volume":"19","author":"Chinchor N","year":"1993"},{"key":"bibr6-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1145\/1297797.1297822"},{"key":"bibr7-1473871613490290","unstructured":"Face recognition grand challenge, http:\/\/www.nist.gov\/itl\/iad\/ig\/frgc.cfm (accessed 8 November 2012)."},{"key":"bibr8-1473871613490290","volume-title":"Contextual design: defining customer-centered systems","author":"Beyer H","year":"1997"},{"key":"bibr9-1473871613490290","volume-title":"Designing the user interface: strategies for effective human-computer interaction","author":"Shneiderman B","year":"2009","edition":"5"},{"key":"bibr10-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2007.70412"},{"key":"bibr11-1473871613490290","unstructured":"NSF International Science & Engineering Visualization Challenge, http:\/\/www.nsf.gov\/news\/special_reports\/scivis\/challenge.jsp (accessed November 2012)."},{"key":"bibr12-1473871613490290","unstructured":"Data Viz Challenge,http:\/\/datavizchallenge.org\/about-challenge (2011, accessed 8 November 2012)."},{"key":"bibr13-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2006.85"},{"key":"bibr14-1473871613490290","first-page":"296","volume-title":"Proceedings of the 38th IEEE annual Hawaii international conference","author":"Cowley PJ"},{"issue":"3","key":"bibr15-1473871613490290","first-page":"102","volume":"2","author":"Redish J.","year":"2007","journal-title":"J Usability Stud"},{"key":"bibr16-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1057\/ivs.2009.16"},{"key":"bibr17-1473871613490290","unstructured":"Netflix Prize,http:\/\/www.netflixprize.com (accessed 8 November 2012)."},{"key":"bibr18-1473871613490290","unstructured":"ODNI Directive 203, http:\/\/www.fas.org\/irp\/dni\/icd\/icd-203.pdf (accessed 20 March 2010)."},{"key":"bibr19-1473871613490290","volume-title":"Structured analytic techniques for intelligence analysis","author":"Heuer R","year":"2010"},{"key":"bibr20-1473871613490290","volume-title":"The thinker\u2019s toolkit: 14 powerful techniques for problem solving","author":"Jones MD.","year":"1998"},{"key":"bibr21-1473871613490290","unstructured":"Apple guidelines, http:\/\/developer.apple.com\/library\/ios\/#DOCUMENTATION\/UserExperience\/Conceptual\/MobileHIG\/Introduction\/Introduction.html (accessed November 2012)."},{"key":"bibr22-1473871613490290","unstructured":"National Cancer Institute. Research-based web design and usability guidelines. Department of Health and Human Services, National Institutes of Health, http:\/\/guidelines.usability.gov\/ (2006, accessed 8 November 2012)."},{"key":"bibr23-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1177\/1473871611407399"},{"key":"bibr24-1473871613490290","volume-title":"Proceedings of the 1999 Ericsson conference on usability engineering, ECUE \u201999","author":"Carr D."},{"key":"bibr25-1473871613490290","first-page":"1","volume-title":"The Proceedings of BELIV \u201810: BEyond time and errors: novel evaLuation methods for Information Visualization","author":"Scholtz J."},{"key":"bibr26-1473871613490290","first-page":"1","volume-title":"Proceedings of BELIV \u201812: BEyond time and errors: novel evaLuation methods for Information Visualization, a workshop of the IEEE VisWeek 2012 conference","author":"Scholtz J"},{"key":"bibr27-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2009.111"},{"key":"bibr28-1473871613490290","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2010.84"},{"key":"bibr29-1473871613490290","first-page":"115","volume-title":"Visual Analytics Science and Technology, 2009. VAST 2009, IEEE Symposium on","author":"Whiting M"}],"container-title":["Information Visualization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1473871613490290","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1473871613490290","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1473871613490290","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T19:18:36Z","timestamp":1777490316000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1473871613490290"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,6,11]]},"references-count":29,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2014,10]]}},"alternative-id":["10.1177\/1473871613490290"],"URL":"https:\/\/doi.org\/10.1177\/1473871613490290","relation":{},"ISSN":["1473-8716","1473-8724"],"issn-type":[{"value":"1473-8716","type":"print"},{"value":"1473-8724","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,6,11]]}}}