{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T22:22:11Z","timestamp":1773267731036,"version":"3.50.1"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2019,2,14]],"date-time":"2019-02-14T00:00:00Z","timestamp":1550102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"American Medical Informatics Association"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n\u2009=\u200943), and the most common data gathering methods were direct observation (n\u2009=\u200930) and surveys (n\u2009=\u200927). About half of the publications focused on static, concentrated views of data with visuals (n\u2009=\u200936). Evaluations were heterogeneous regarding setting and measurements used.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocy190","type":"journal-article","created":{"date-parts":[[2018,12,21]],"date-time":"2018-12-21T13:37:41Z","timestamp":1545399461000},"page":"314-323","source":"Crossref","is-referenced-by-count":42,"title":["Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation"],"prefix":"10.1093","volume":"26","author":[{"given":"Danny T Y","family":"Wu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, USA"}]},{"given":"Annie T","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, Seattle, Washington, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2257-6556","authenticated-orcid":false,"given":"John D","family":"Manning","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Atrium Health's Carolinas Medical Center, Charlotte, North Carolina, USA"}]},{"given":"Gal","family":"Levy-Fix","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}]},{"given":"Uba","family":"Backonja","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, Seattle, Washington, USA"},{"name":"Nursing & Healthcare Leadership, University of Washington Tacoma, Tacoma, Washington"}]},{"given":"David","family":"Borland","sequence":"additional","affiliation":[{"name":"Renaissance Computing Institute, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"}]},{"given":"Jesus J","family":"Caban","sequence":"additional","affiliation":[{"name":"National Intrepid Center of Excellence, Walter Reed National Military Medical Center, Bethesda, Maryland, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5672-8605","authenticated-orcid":false,"given":"Dawn W","family":"Dowding","sequence":"additional","affiliation":[{"name":"Division of Nursing, Midwifery and Social Work, School of Health Sciences, University of Manchester, Manchester, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8793-9982","authenticated-orcid":false,"given":"Harry","family":"Hochheiser","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Intelligent Systems Program, University of Pittsburgh, Pittsburgh, Pennsylvania, USA"}]},{"given":"Vadim","family":"Kagan","sequence":"additional","affiliation":[{"name":"SentiMetrix, Inc, Bethesda, Maryland, USA"}]},{"given":"Swaminathan","family":"Kandaswamy","sequence":"additional","affiliation":[{"name":"Department of Mechanical and Industrial Engineering, University of Massachusetts at Amherst, Amherst, Massachusetts, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2207-329X","authenticated-orcid":false,"given":"Manish","family":"Kumar","sequence":"additional","affiliation":[{"name":"MEASURE Evaluation, Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"},{"name":"Carolina Health Informatics Program, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"}]},{"given":"Alexis","family":"Nunez","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0667-9061","authenticated-orcid":false,"given":"Eric","family":"Pan","sequence":"additional","affiliation":[{"name":"Healthcare Delivery Research and Evaluation, Westat, Rockville, Maryland, USA"}]},{"given":"David","family":"Gotz","sequence":"additional","affiliation":[{"name":"Carolina Health Informatics Program, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"},{"name":"School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA"}]}],"member":"286","published-online":{"date-parts":[[2019,2,14]]},"reference":[{"issue":"2","key":"2020110613060604700_ocy190-B1","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1093\/jamia\/ocv006","article-title":"Visual analytics in healthcare\u2013opportunities and research challenges","volume":"22","author":"Caban","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020110613060604700_ocy190-B2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/MCG.2016.59","article-title":"Data-driven healthcare: challenges and opportunities for interactive visualization","volume":"36","author":"Gotz","year":"2016","journal-title":"IEEE Comput Grap Appl"},{"issue":"6","key":"2020110613060604700_ocy190-B3","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1145\/1743546.1743567","article-title":"A tour through the visualization zoo","volume":"53","author":"Heer","year":"2010","journal-title":"Commun ACM"},{"key":"2020110613060604700_ocy190-B4","volume-title":"Readings in Information Visualization: Using Vision to Think","author":"Card","year":"1999"},{"key":"2020110613060604700_ocy190-B5","volume-title":"Illuminating the Path: The Research and Development Agenda for Visual Analytics","author":"Thomas","year":"2005"},{"issue":"9","key":"2020110613060604700_ocy190-B6","doi-asserted-by":"crossref","first-page":"1265, 1269.","DOI":"10.1001\/jama.1990.03440090103036","article-title":"Practice policies: where do they come from?","volume":"263","author":"Eddy","year":"1990","journal-title":"JAMA"},{"key":"2020110613060604700_ocy190-B7"},{"key":"2020110613060604700_ocy190-B8"},{"key":"2020110613060604700_ocy190-B9"},{"issue":"4","key":"2020110613060604700_ocy190-B10","doi-asserted-by":"crossref","first-page":"e101","DOI":"10.2196\/mhealth.4924","article-title":"Health app use among us mobile phone owners: a national survey","volume":"3","author":"Krebs","year":"2015","journal-title":"JMIR Mhealth Uhealth"},{"key":"2020110613060604700_ocy190-B11"},{"issue":"2","key":"2020110613060604700_ocy190-B12","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1136\/amiajnl-2014-002955","article-title":"Innovative information visualization of electronic health record data: a systematic review","volume":"22","author":"West","year":"2015","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613060604700_ocy190-B13"},{"issue":"4","key":"2020110613060604700_ocy190-B14","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1093\/jamia\/ocx126","article-title":"Development of the PHAST model: generating standard public health services data and evidence for decision-making","volume":"25","author":"Bekemeier","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613060604700_ocy190-B15","author":"Shneiderman"},{"key":"2020110613060604700_ocy190-B16","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1007\/978-3-540-70956-5_2","volume-title":"Information Visualization: Human-Centered Issues and Perspectives","author":"Carpendale","year":"2008"},{"issue":"3","key":"2020110613060604700_ocy190-B17","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/MCG.2006.70","article-title":"Towards measuring visualization insight","volume":"26","author":"North","year":"2006","journal-title":"IEEE Comput Grap Appl"},{"issue":"6","key":"2020110613060604700_ocy190-B18","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1109\/TVCG.2009.111","article-title":"A nested model for visualization design and validation","volume":"15","author":"Munzner","year":"2009","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"2020110613060604700_ocy190-B19","author":"Meyer","year":"2012"},{"issue":"3","key":"2020110613060604700_ocy190-B20","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1177\/1473871613510429","article-title":"The nested blocks and guidelines model","volume":"14","author":"Meyer","year":"2015","journal-title":"Inf Vis"},{"issue":"1","key":"2020110613060604700_ocy190-B21","first-page":"i","article-title":"User-centered evaluation of visual analytics","volume":"5","author":"Scholtz","year":"2017","journal-title":"Synth Lect Vis"},{"issue":"1","key":"2020110613060604700_ocy190-B22","doi-asserted-by":"crossref","first-page":"b2700","DOI":"10.1136\/bmj.b2700","article-title":"The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration","volume":"339","author":"Liberati","year":"2009","journal-title":"BMJ"},{"issue":"1","key":"2020110613060604700_ocy190-B23","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ Psychol Meas"},{"issue":"9","key":"2020110613060604700_ocy190-B24","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1177\/1049732305276687","article-title":"Three approaches to qualitative content analysis","volume":"15","author":"Hsieh","year":"2005","journal-title":"Qual Health Res"},{"key":"2020110613060604700_ocy190-B25","volume-title":"Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0","author":"Higgins","year":"2011"},{"key":"2020110613060604700_ocy190-B26","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1109\/VL.1996.545307","volume-title":"Proceedings 1996 IEEE Symposium on Visual Languages","author":"Shneiderman","year":"1996"},{"key":"2020110613060604700_ocy190-B27","author":"Zoss","year":"2012"},{"key":"2020110613060604700_ocy190-B28"},{"key":"2020110613060604700_ocy190-B29","author":"Gotz","year":"2016"},{"issue":"1","key":"2020110613060604700_ocy190-B30","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"2020110613060604700_ocy190-B31","volume-title":"Evaluation Methods in Biomedical Informatics","author":"Friedman","year":"2005"},{"issue":"3","key":"2020110613060604700_ocy190-B32","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1109\/MCG.2009.56","article-title":"Visual-analytics evaluation","volume":"29","author":"Plaisant","year":"2009","journal-title":"IEEE Comput Graph Appl"},{"issue":"7","key":"2020110613060604700_ocy190-B33","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1109\/MC.2013.151","article-title":"Evaluation: a challenge for visual analytics","volume":"46","author":"van Wijk","year":"2013","journal-title":"Computer"},{"issue":"4","key":"2020110613060604700_ocy190-B34","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1038\/ng1201-365","article-title":"Minimum information about a microarray experiment (MIAME)-toward standards for microarray data","volume":"29","author":"Brazma","year":"2001","journal-title":"Nat Genet"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/4\/314\/34151495\/ocy190.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/4\/314\/34151495\/ocy190.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,19]],"date-time":"2020-11-19T01:35:53Z","timestamp":1605749753000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/4\/314\/5320044"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,2,14]]},"references-count":34,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2019,2,14]]},"published-print":{"date-parts":[[2019,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy190","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,4]]},"published":{"date-parts":[[2019,2,14]]}}}