{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T17:24:38Z","timestamp":1776273878854,"version":"3.50.1"},"reference-count":105,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2024,12,28]],"date-time":"2024-12-28T00:00:00Z","timestamp":1735344000000},"content-version":"vor","delay-in-days":366,"URL":"http:\/\/www.sagepub.com\/licence-information-for-chorus"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["LE220100078 and DP220103717"],"award-info":[{"award-number":["LE220100078 and DP220103717"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Information Visualization"],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:p> In recent times, visual analytics systems (VAS) have been used to solve various complex issues in diverse application domains. Nonetheless, an inherent drawback arises from the insufficient evaluation of VAS, resulting in occasional inaccuracies when it comes to analytical reasoning, information synthesis, and deriving insights from vast, ever-changing, ambiguous, and frequently contradictory data. Hence, the significance of implementing an appropriate evaluation methodology cannot be overstated, as it plays a pivotal role in enhancing the design and development of visualization systems. This paper assesses visualization systems by providing a systematic exploration of various evaluation strategies (ES). While several existing studies have examined some ES, the extent of comprehensive and systematic review for visualization research remains limited. In this work, we introduce seven state-of-the-art and widely recognized ES namely (1) dashboard comparison; (2) insight-based evaluation; (3) log data analysis; (4) Likert scales; (5) qualitative and quantitative analysis; (6) Nielsen\u2019s heuristics; and (7) eye trackers. Moreover, it delves into their historical context and explores numerous applications where these ES have been employed, shedding light on the associated evaluation practices. Through our comprehensive review, we overview and analyze the predominant evaluation goals within the visualization community, elucidating their evolution and the inherent contrasts. Additionally, we identify the open challenges that arise with the emergence of new ES, while also highlighting the key themes gleaned from the existing literature that hold potential for further exploration in future studies. <\/jats:p>","DOI":"10.1177\/14738716231212568","type":"journal-article","created":{"date-parts":[[2023,12,29]],"date-time":"2023-12-29T07:27:00Z","timestamp":1703834820000},"page":"84-101","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":11,"title":["Strategies for evaluating visual analytics systems: A systematic review and new perspectives"],"prefix":"10.1177","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7209-3881","authenticated-orcid":false,"given":"Md Rafiqul","family":"Islam","sequence":"first","affiliation":[{"name":"Data Science Institute, University of Technology Sydney, Broadway, NSW, Australia"}]},{"given":"Shanjita","family":"Akter","sequence":"additional","affiliation":[{"name":"School of Computer Science, Taylors University, Subang Jaya, Selangor, Malaysia"}]},{"given":"Linta","family":"Islam","sequence":"additional","affiliation":[{"name":"Department of Biological Science, Louisiana State University, Baton Rouge, LA, USA"}]},{"given":"Imran","family":"Razzak","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, UNSW, Sydney, NSW, Australia"}]},{"given":"Xianzhi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Technology Sydney, Ultimo, NSW, Australia"}]},{"given":"Guandong","family":"Xu","sequence":"additional","affiliation":[{"name":"Data Science Institute, University of Technology Sydney, Broadway, NSW, Australia"}]}],"member":"179","published-online":{"date-parts":[[2023,12,28]]},"reference":[{"key":"bibr1-14738716231212568","first-page":"7","volume-title":"Adjunct Proceedings of the 2021 International Conference on Distributed Computing and Networking","author":"Mandal A"},{"key":"bibr2-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2006.85"},{"key":"bibr3-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-70956-5_7"},{"key":"bibr4-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocv006"},{"key":"bibr5-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2011.279"},{"key":"bibr6-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.01.105"},{"key":"bibr7-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2018.2881501"},{"key":"bibr8-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2006.5"},{"key":"bibr9-14738716231212568","first-page":"1","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems 2021","author":"Li H"},{"key":"bibr10-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"bibr11-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2007.70412"},{"key":"bibr12-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2640960"},{"key":"bibr13-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/B978-155860915-0\/50046-9"},{"key":"bibr14-14738716231212568","doi-asserted-by":"publisher","DOI":"10.3390\/electronics11091344"},{"key":"bibr15-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744718"},{"key":"bibr16-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598796"},{"key":"bibr17-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2005.63"},{"key":"bibr18-14738716231212568","volume-title":"Illuminating the path: The research and development agenda for visual analytics","author":"Cook KA"},{"key":"bibr19-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1089\/big.2015.0059"},{"key":"bibr20-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/IV.2007.130"},{"key":"bibr21-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-013-0892-3"},{"key":"bibr22-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2003.1210860"},{"key":"bibr23-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13210"},{"key":"bibr24-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1177\/14738716221085965"},{"key":"bibr25-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2018.1466266"},{"key":"bibr26-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.126"},{"key":"bibr27-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/1168149.1168151"},{"key":"bibr28-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-019-04372-x"},{"key":"bibr29-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2000.0422"},{"key":"bibr30-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/IV.2010.33"},{"key":"bibr31-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/j.visinf.2017.01.006"},{"key":"bibr32-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2008.4677361"},{"key":"bibr33-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365964"},{"key":"bibr34-14738716231212568","first-page":"1","volume-title":"2021 International Joint Conference on Neural Networks (IJCNN)","author":"Islam MR","year":"2021"},{"key":"bibr35-14738716231212568","volume-title":"The 23rd International Conference on Computer and Information Technology (ICCIT-2020)","author":"Islam MT"},{"key":"bibr36-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864504"},{"key":"bibr37-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2013.54"},{"key":"bibr38-14738716231212568","first-page":"228","author":"Abdulla N","year":"2015","journal-title":"Int J Innov Res Sci Technol"},{"key":"bibr39-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2020.102294"},{"key":"bibr40-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1287\/serv.2018.0210"},{"key":"bibr41-14738716231212568","first-page":"7541","volume":"13","author":"Almaimoni H","year":"2018","journal-title":"Int J Appl Sci Eng Res"},{"issue":"9","key":"bibr42-14738716231212568","volume-title":"Challenges in the adoption of visualization system: a survey","volume":"37","author":"Folorunso O","year":"2008"},{"key":"bibr43-14738716231212568","unstructured":"Geisler G. Making information more accessible: A survey of information visualization applications and techniques, http:\/\/www.ils.unc.edu\/\u223cgeisg\/info\/infovis\/paper.html (1998, accessed 31 January)."},{"key":"bibr44-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2016.7591442"},{"key":"bibr45-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/INFVIS.2004.5"},{"key":"bibr46-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1055\/s-0038-1666842"},{"key":"bibr47-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888905000299"},{"key":"bibr48-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/2669557.2669560"},{"key":"bibr49-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/s12650-015-0342-6"},{"key":"bibr50-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-020-09866-x"},{"key":"bibr51-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/NOF.2018.8598127"},{"key":"bibr52-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijsu.2021.105906"},{"key":"bibr53-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/1377966.1377978"},{"key":"bibr54-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865146"},{"key":"bibr55-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1177\/1473871615609787"},{"key":"bibr56-14738716231212568","first-page":"913916","volume-title":"2016 IEEE 14th Intl Conf on Dependable, Autonomic and Secure Computing, 14th Intl Conf on Pervasive Intelligence and Computing, 2nd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC\/PiCom\/DataCom\/CyberSciTech)","author":"Leung C"},{"key":"bibr57-14738716231212568","first-page":"517","volume-title":"2016 IEEE International Conference on Healthcare Informatics (ICHI)","author":"Kamaleswaran R"},{"key":"bibr58-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2467871"},{"key":"bibr59-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-021-10972-3"},{"key":"bibr60-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/1168149.1168162"},{"key":"bibr61-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467224"},{"key":"bibr62-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/IV53921.2021.00018"},{"key":"bibr63-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/CCWC47524.2020.9031237"},{"key":"bibr64-14738716231212568","first-page":"12316","volume":"2009","author":"Qian X","year":"2020","journal-title":"arXiv preprint arXiv"},{"key":"bibr65-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1080\/13658816.2019.1594823"},{"key":"bibr66-14738716231212568","doi-asserted-by":"publisher","DOI":"10.3390\/info11060314"},{"key":"bibr67-14738716231212568","doi-asserted-by":"publisher","DOI":"10.2196\/20645"},{"key":"bibr68-14738716231212568","volume":"2002","author":"Webster J","year":"2002","journal-title":"MIS Q"},{"key":"bibr69-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/PacificVis48177.2020.1261"},{"key":"bibr70-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2922597"},{"key":"bibr71-14738716231212568","author":"Bourqui R","year":"2019","journal-title":"ArXiv"},{"key":"bibr72-14738716231212568","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2019.634"},{"key":"bibr73-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22643-5_18"},{"key":"bibr74-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2019.2924636"},{"key":"bibr75-14738716231212568","first-page":"1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Com-puting Systems","author":"Hu K","year":"2019"},{"key":"bibr76-14738716231212568","first-page":"800","volume":"26","author":"Deng Z","year":"2020","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"bibr77-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934668"},{"key":"bibr78-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13667"},{"key":"bibr79-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/iV.2018.00038"},{"key":"bibr80-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865027"},{"key":"bibr81-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/PacificVis.2018.00033"},{"key":"bibr82-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/PacificVis.2018.00036"},{"key":"bibr83-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/KELVAR.2017.7961559"},{"key":"bibr84-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2017.2762673"},{"key":"bibr85-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/3038462.3038463"},{"key":"bibr86-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2017.8585721"},{"key":"bibr87-14738716231212568","first-page":"471","volume-title":"International Conference on Applied Human Factors and Ergonomics","author":"Swaid S","year":"2017"},{"key":"bibr88-14738716231212568","first-page":"61","volume-title":"2016 IEEE Conference on Visual Analytics Science and Technology (VAST)","author":"Du F"},{"key":"bibr89-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2011.08.030"},{"key":"bibr90-14738716231212568","unstructured":"Z\u00f6ller MA, Titov W, Schlegel T, et al. XAutoML: A Visual Analytics Tool for Establishing Trust in Automated ML. arXiv preprint arXiv:2202.11954, 2022."},{"key":"bibr91-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2009.22"},{"key":"bibr92-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2018.06.015"},{"key":"bibr93-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/69.553159"},{"key":"bibr94-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2006.70"},{"key":"bibr95-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2005.53"},{"key":"bibr96-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/3092931.3092937"},{"key":"bibr97-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/BESC51023.2020.9348301"},{"key":"bibr98-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"key":"bibr99-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744158"},{"key":"bibr100-14738716231212568","volume-title":"Brazilian Symposium on Human Factors in Computing Systems","author":"Dal CM","year":"2002"},{"key":"bibr101-14738716231212568","volume-title":"Proceedings of the 2016 IEEE Conference on Information Visualization","author":"Silva N","year":"2016"},{"key":"bibr102-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1155\/2016\/9172506"},{"key":"bibr103-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1179\/000870408X311396"},{"key":"bibr104-14738716231212568","doi-asserted-by":"publisher","DOI":"10.1145\/1168149.1168158"},{"key":"bibr105-14738716231212568","doi-asserted-by":"publisher","DOI":"10.2991\/hcis.k.210704.003"}],"container-title":["Information Visualization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716231212568","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/14738716231212568","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716231212568","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716231212568","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T19:21:10Z","timestamp":1740943270000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/14738716231212568"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,28]]},"references-count":105,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1]]}},"alternative-id":["10.1177\/14738716231212568"],"URL":"https:\/\/doi.org\/10.1177\/14738716231212568","relation":{},"ISSN":["1473-8716","1473-8724"],"issn-type":[{"value":"1473-8716","type":"print"},{"value":"1473-8724","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,28]]}}}